Tasks stuck with "waiting for Pulp to start the task" - foreman-2.4 / Katello 4.0.0 / Pulp3

laimsi · May 12, 2021, 10:12am

Hello,

I deployed fresh server with foreman-2.4 / Katello 4.0.0 / Pulp3.

After deployment I tryed to add some repositories CV, CVV but tasks are stuck for more then 15 hours with message : (waiting for Pulp to start the task).

How can I identify on which task or sub-task Pulp is blocked?
Is there a way to reset pulp tasks or a cleanup?

I tryed to restart all foreman-maintain service with foreman-maintain service restart and also restart the server several time but still the same status of stuck taks.

With Pulp3 we can’t use pulp-admin tool any more, so how can I unblock the situation and make tasks running successfully again?

Regards,

laimsi · May 12, 2021, 11:06am

Below output of curl https://foremanserver/pulp/api/v3/status/

[root@foremanserver ~]# curl https://foremanserver/pulp/api/v3/status/ | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2328  100  2328    0     0  16202      0 --:--:-- --:--:-- --:--:-- 16279
{
  "versions": [
    {
      "component": "pulpcore",
      "version": "3.9.1"
    },
    {
      "component": "pulp_rpm",
      "version": "3.10.0"
    },
    {
      "component": "pulp_file",
      "version": "1.5.0"
    },
    {
      "component": "pulp_deb",
      "version": "2.9.1"
    },
    {
      "component": "pulp_container",
      "version": "2.2.1"
    },
    {
      "component": "pulp_certguard",
      "version": "1.1.0"
    }
  ],
  "online_workers": [
    {
      "pulp_created": "2021-05-10T19:48:14.375531Z",
      "pulp_href": "/pulp/api/v3/workers/32cbe404-fd67-4e7d-a248-0382bac60c0c/",
      "name": "resource-manager",
      "last_heartbeat": "2021-05-12T11:02:03.972287Z"
    },
    {
      "pulp_created": "2021-05-12T10:15:25.442660Z",
      "pulp_href": "/pulp/api/v3/workers/1a990a41-af81-4d21-b8c6-ba163e1dff05/",
      "name": "18439@foremanserver",
      "last_heartbeat": "2021-05-12T11:02:11.386376Z"
    },
    {
      "pulp_created": "2021-05-12T10:15:27.533365Z",
      "pulp_href": "/pulp/api/v3/workers/4899e382-d255-45ba-8ee7-c33a8f9733eb/",
      "name": "18440@foremanserver",
      "last_heartbeat": "2021-05-12T11:02:13.890583Z"
    },
    {
      "pulp_created": "2021-05-12T10:15:28.240457Z",
      "pulp_href": "/pulp/api/v3/workers/5273f427-20ce-4a89-a5ac-76cf6d0644f9/",
      "name": "18441@foremanserver",
      "last_heartbeat": "2021-05-12T11:02:14.290996Z"
    },
    {
      "pulp_created": "2021-05-12T10:15:28.605294Z",
      "pulp_href": "/pulp/api/v3/workers/8989bba9-aeab-409a-baf0-ab0ac42d0340/",
      "name": "18431@foremanserver",
      "last_heartbeat": "2021-05-12T11:02:15.393839Z"
    }
  ],
  "online_content_apps": [
    {
      "name": "18405@foremanserver",
      "last_heartbeat": "2021-05-12T11:03:05.929437Z"
    },
    {
      "name": "18338@foremanserver",
      "last_heartbeat": "2021-05-12T11:03:05.929986Z"
    },
    {
      "name": "18384@foremanserver",
      "last_heartbeat": "2021-05-12T11:03:09.539088Z"
    },
    {
      "name": "18419@foremanserver",
      "last_heartbeat": "2021-05-12T11:03:11.504469Z"
    },
    {
      "name": "18319@foremanserver",
      "last_heartbeat": "2021-05-12T11:03:11.710046Z"
    },
    {
      "name": "18363@foremanserver",
      "last_heartbeat": "2021-05-12T11:03:11.903433Z"
    },
    {
      "name": "18400@foremanserver",
      "last_heartbeat": "2021-05-12T11:03:05.279501Z"
    },
    {
      "name": "18425@foremanserver",
      "last_heartbeat": "2021-05-12T11:03:05.580242Z"
    },
    {
      "name": "18308@foremanserver",
      "last_heartbeat": "2021-05-12T11:03:05.810499Z"
    }
  ],
  "database_connection": {
    "connected": true
  },
  "redis_connection": {
    "connected": true
  },
  "storage": {
    "total": 214695936000,
    "used": 122678865920,
    "free": 92017070080
  }
}

laimsi · May 12, 2021, 11:14am

Hello ehelms & @ evgeni do you have any idea about this pulp3 and tasks issue?

Thanks for your help.

iballou · May 12, 2021, 6:22pm

Hi @laimsi,

It looks like you probably have an orphaned reserved-resource worker in Pulp 3. It’s a bug that has popped up for some people in the past. The Pulp 3 team has a document for dealing with it. Please try the steps here, in the section I’m linking: Reserved-Resource debugging - HackMD

Keep in mind the Katello caveats that are listed right above the section I linked. pulpcore-manager needs to be run like so from a directory that the pulp user can read and write to (like /tmp for example):

sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager ...

laimsi · May 12, 2021, 7:39pm

Hello @iballou,

Thanks for your answer, I executed command below but looks like there is no reserved-resource worker

[root@formanserver ~]# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell <<EOF
> from pulpcore.app.models import ReservedResource, Worker
> worker_to_res = {}
> for rr in ReservedResource.objects.all():
>   worker_to_res[rr.worker_id] = rr.pulp_id
> workers = [w.pulp_id for w in Worker.objects.online_workers()]
> for rwork in worker_to_res:
>   if rwork not in workers:
>     print(f'Worker {rwork} owns ReservedResource {worker_to_res[rwork]} and is not in online_workers!!')
> EOF
[root@formanserver ~]#
[root@formanserver ~]#

Below also result of pulp tasks in waiting status :

[root@formanserver ~]# curl https://`hostname`/pulp/api/v3/tasks/?state=waiting --cert /etc/pki/katello/certs/pulp-client.crt  --key /etc/pki/katello/private/pulp-client.key | jq
{
  "count": 18,
  "next": null,
  "previous": null,
  "results": [
    {
      "pulp_href": "/pulp/api/v3/tasks/ae84cabb-1531-4192-9b48-be973bf9f3e2/",
      "pulp_created": "2021-05-12T18:53:35.937847Z",
      "state": "waiting",
      "name": "pulp_deb.app.tasks.publishing.publish",
      "logging_cid": "5a002c96598f45b7aea925a79fda9d9b",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/827815db-7f88-4d77-89e1-d351a2bce120/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/042def9d-550b-4eb0-af31-ddfb7c7fa893/",
      "pulp_created": "2021-05-12T18:53:35.599286Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_update",
      "logging_cid": "72c6e943d2d4461d8d25e24449998172",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/45a584ae-3e51-45ab-8e33-aa003f84a91e/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/43e6ac99-fedb-416a-acbc-5872bf7113fb/",
      "pulp_created": "2021-05-12T18:53:34.914631Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_update",
      "logging_cid": "c28d58d04a054293a382f00880e3260b",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/remotes/deb/apt/cbe41858-7231-4b76-b292-71b1730cbfa3/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/a8d6c790-2d18-41bd-b7d4-892266d34584/",
      "pulp_created": "2021-05-12T18:53:34.473959Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_update",
      "logging_cid": "679c6e77133c417fb9f5437447b654ee",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/827815db-7f88-4d77-89e1-d351a2bce120/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/1a6fd4b9-4c8e-4f79-8957-3406da0bdbb0/",
      "pulp_created": "2021-05-12T09:49:22.053688Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "cda05c77fc4d4a4fb17a05e1be8b0435",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/9291001d-ec28-4fb6-96ad-264611c64f69/",
      "pulp_created": "2021-05-12T09:49:22.017887Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "10d768d618ea4c4a96b0eb45c2218634",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/9e520a02-6819-4b6b-a919-d0a98ae08878/",
      "pulp_created": "2021-05-12T09:49:21.978039Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "115a3110ccb54ab3a86ed222daea3915",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/b79c5fef-3333-4afb-8249-b4ea5787daac/",
      "pulp_created": "2021-05-12T09:49:21.937124Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "703886c2909943a5adfa05a82795602a",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/63fbb419-3c1e-4c8f-8d10-b604baf00d1e/",
      "pulp_created": "2021-05-12T09:49:00.128477Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "3c48339155b044618dd6bda7d825c447",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/45a584ae-3e51-45ab-8e33-aa003f84a91e/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/c4e0d43b-176c-4a26-a386-75cf85cde6b6/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/926775aa-4f59-46bd-877e-5c30d580661e/",
      "pulp_created": "2021-05-12T09:49:00.047113Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "938a6f4729ab477eb1de1d172650291e",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/ea23faa4-eb56-4d1b-bdc2-d9779fb91051/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/3c71dbc0-936b-47fd-ae31-12ddcbaf9377/",
      "pulp_created": "2021-05-12T09:48:59.913419Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "d110524ce2e348cf9a878e9117bfd1a6",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/ccfbef95-5387-467d-99e5-420ced8dedbf/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/3bed5208-06f1-44a2-9b98-ab449f231a41/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/4472cad9-8390-46c7-a9ba-3dbe06689049/",
      "pulp_created": "2021-05-12T09:48:42.350770Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "c9d9b62b4ff340a1b805346656573ada",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/107b4468-f579-4e2a-8939-70251f98150f/",
      "pulp_created": "2021-05-12T09:48:42.318951Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "de888e671f7444d392395acb40a3bdc3",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/c256f1a8-3be8-4fd2-a191-051c2d186b9b/",
      "pulp_created": "2021-05-12T09:48:42.285439Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "770a6a7d48e74080aebeb66ae6591783",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/f665b048-c19b-455b-a90d-06dde3c3c71c/",
      "pulp_created": "2021-05-12T09:48:15.892560Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "cd75feb1de3341e48bd429408577a0b1",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/8a756286-c136-485c-a412-df48c43d47aa/",
      "pulp_created": "2021-05-12T09:48:15.858513Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "efcc734a843e4589983299a8257bbede",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/34b6bd43-ea9d-43ef-9332-d18dae566546/",
      "pulp_created": "2021-05-12T09:48:15.826673Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "39b5eff1df4741c1abdc65418c01f1ed",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/f5505640-03aa-44d5-9c73-b40a919d5102/",
      "pulp_created": "2021-05-12T09:48:15.717927Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "d04844a051ca49d1ac120e8d7d9f3e8e",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    }
  ]
}

Here liste of pulp workers :

[root@formanserver ~]# curl https://`hostname`/pulp/api/v3/workers/  --cert /etc/pki/katello/certs/pulp-client.crt  --key /etc/pki/katello/private/pulp-client.key  | jq
{
  "count": 17,
  "next": null,
  "previous": null,
  "results": [
    {
      "pulp_created": "2021-05-12T19:08:23.327781Z",
      "pulp_href": "/pulp/api/v3/workers/9f17f8b0-d442-40a6-8501-060ea06368d5/",
      "name": "8635@foremanserver",
      "last_heartbeat": "2021-05-12T19:34:52.903404Z"
    },
    {
      "pulp_created": "2021-05-12T19:08:23.289755Z",
      "pulp_href": "/pulp/api/v3/workers/26ac8319-e622-4e7e-83f3-680b3902eb27/",
      "name": "8632@foremanserver",
      "last_heartbeat": "2021-05-12T19:34:52.502139Z"
    },
    {
      "pulp_created": "2021-05-12T19:08:22.862526Z",
      "pulp_href": "/pulp/api/v3/workers/2971eb63-fdea-4ba3-9a38-bc68deede1e0/",
      "name": "8633@foremanserver",
      "last_heartbeat": "2021-05-12T19:35:09.129404Z"
    },
    {
      "pulp_created": "2021-05-12T19:08:22.333356Z",
      "pulp_href": "/pulp/api/v3/workers/341b23e0-30d3-4e34-9ab2-edae26b54982/",
      "name": "8634@foremanserver",
      "last_heartbeat": "2021-05-12T19:35:07.428342Z"
    },
    {
      "pulp_created": "2021-05-12T10:15:28.605294Z",
      "pulp_href": "/pulp/api/v3/workers/8989bba9-aeab-409a-baf0-ab0ac42d0340/",
      "name": "18431@foremanserver",
      "last_heartbeat": "2021-05-12T19:13:24.527504Z"
    },
    {
      "pulp_created": "2021-05-12T10:15:28.240457Z",
      "pulp_href": "/pulp/api/v3/workers/5273f427-20ce-4a89-a5ac-76cf6d0644f9/",
      "name": "18441@foremanserver",
      "last_heartbeat": "2021-05-12T19:11:59.079129Z"
    },
    {
      "pulp_created": "2021-05-12T10:15:27.533365Z",
      "pulp_href": "/pulp/api/v3/workers/4899e382-d255-45ba-8ee7-c33a8f9733eb/",
      "name": "18440@foremanserver",
      "last_heartbeat": "2021-05-12T19:13:24.018947Z"
    },
    {
      "pulp_created": "2021-05-12T10:15:25.442660Z",
      "pulp_href": "/pulp/api/v3/workers/1a990a41-af81-4d21-b8c6-ba163e1dff05/",
      "name": "18439@foremanserver",
      "last_heartbeat": "2021-05-12T19:13:23.501735Z"
    },
    {
      "pulp_created": "2021-05-12T09:44:08.915210Z",
      "pulp_href": "/pulp/api/v3/workers/c72bc34e-f372-4376-bd65-c9066690577f/",
      "name": "1713@foremanserver",
      "last_heartbeat": "2021-05-12T10:20:17.923549Z"
    },
    {
      "pulp_created": "2021-05-12T09:44:08.104734Z",
      "pulp_href": "/pulp/api/v3/workers/c5bd1542-6c93-41ee-99f5-c955ee8e563d/",
      "name": "1697@foremanserver",
      "last_heartbeat": "2021-05-12T10:20:17.910476Z"
    },
    {
      "pulp_created": "2021-05-12T09:44:07.870187Z",
      "pulp_href": "/pulp/api/v3/workers/e05038dd-1e54-4cc8-8676-f7b96af455d3/",
      "name": "1709@foremanserver",
      "last_heartbeat": "2021-05-12T10:20:17.919404Z"
    },
    {
      "pulp_created": "2021-05-12T09:44:07.422913Z",
      "pulp_href": "/pulp/api/v3/workers/2e8a38b0-f879-4e15-9bd0-d640c14e9fd2/",
      "name": "1715@foremanserver",
      "last_heartbeat": "2021-05-12T10:20:17.915188Z"
    },
    {
      "pulp_created": "2021-05-10T19:48:14.375531Z",
      "pulp_href": "/pulp/api/v3/workers/32cbe404-fd67-4e7d-a248-0382bac60c0c/",
      "name": "resource-manager",
      "last_heartbeat": "2021-05-12T19:35:09.129392Z"
    },
    {
      "pulp_created": "2021-05-10T19:48:09.909268Z",
      "pulp_href": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "name": "6347@foremanserver",
      "last_heartbeat": "2021-05-12T18:56:45.955889Z"
    },
    {
      "pulp_created": "2021-05-10T19:48:09.629296Z",
      "pulp_href": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "name": "6214@foremanserver",
      "last_heartbeat": "2021-05-12T18:56:45.960076Z"
    },
    {
      "pulp_created": "2021-05-10T19:48:08.906922Z",
      "pulp_href": "/pulp/api/v3/workers/45a584ae-3e51-45ab-8e33-aa003f84a91e/",
      "name": "6086@foremanserver",
      "last_heartbeat": "2021-05-12T18:56:45.964094Z"
    },
    {
      "pulp_created": "2021-05-10T19:48:08.191210Z",
      "pulp_href": "/pulp/api/v3/workers/ccfbef95-5387-467d-99e5-420ced8dedbf/",
      "name": "5955@foremanserver",
      "last_heartbeat": "2021-05-12T18:49:12.252583Z"
    }
  ]
}

iballou · May 12, 2021, 8:40pm

Hey @laimsi,

After talking with some of the Pulp folks, it sounds like there is some sort of data mismatch going on with the Pulpcore database and your system’s workers. We’re not sure what would cause it and would need to take a closer look at your system’s database and processes.

From my perspective, since you said your Katello is fresh, would you be okay with simply doing a full reset (foreman-installer --reset-data) ? Dumping and re-creating the Pulpcore database (which the reset would do) would almost definitely solve your issue. If it doesn’t, then at least we’d have a reproducible bug.

If you can’t reset your system, let us know. We’d need to come up with some new commands to run to get the necessary debugging info from your system.

laimsi · May 13, 2021, 11:22am

Hello @iballou,

I have 2 Forman/katello servers one for test and second for prod environnement .
I’m facing tasks stuck issue on both environnement (test and prod)

I tryed foreman-installer --reset-data on test server and this reset all data on my foreman test server and delete all created resources (products, activation key, repositories, CV, CCV, hosts… ).

In production environnement I can’t use –reset-data and delete all data on production.

So if you can come up with some new commands to run and get the necessary debugging info from my prod system.

Thanks for your help

Regards,

iballou · May 13, 2021, 12:27pm

@laimsi did resetting your test server fix the stuck tasks problem?

I’m surprised to hear you’re hitting this issue on both systems. Are they completely separate fresh installs, or was one cloned/copied from another?

ricar · May 13, 2021, 1:35pm

Hello @iballou @laimsi,

I have a simmilar issue on Repo sync fails after a first app restart on new installation of Foreman 2.4 / Katello 4.0

As it was a fleshly installation too, i’m testing " [Fixes #32475 - clean_backend_objects still uses Pulp 2]" + “foreman-installer --reset-data” solution.

For the momment, that fix for me.

Regards.

iballou · May 13, 2021, 1:40pm

Thanks for the information @ricar. Pulp 3 is moving to a completely new tasking system sometime soon so that will hopefully fix all of these issues.

laimsi · May 13, 2021, 2:15pm

Hello @iballou ,

Yes resetting my test server fix the stuck tasks problem, but this action reset all data on my test server and delete all created resources (products, activation key, repositories, CV, CCV, hosts… ).

These 2 servers are completely separate fresh installs

Regards,

iballou · May 13, 2021, 3:33pm

@laimsi the next debugging steps are unfamiliar to me so we’ll need support from the Pulp team. I’ve pinged them on IRC but they’re pretty busy. You might want to subscribe to the pulp-dev list and send an email with the stuck task information above. Here’s the email list info: Pulp-dev Info Page

iballou · May 13, 2021, 3:58pm

@laimsi the Pulp team gave me something else to try:

systemctl stop pulpcore* --all’
cd /tmp

sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell -c "import pulpcore; pulpcore.app.models.ReservedResource.objects.all().delete()"

systemctl restart pulpcore* --all

Make sure the Pulpcore services are all stopped (step 1) because otherwise it could be a dangerous operation.

laimsi · May 13, 2021, 5:20pm

Hello @iballou

I executed the 3 commands as below :

[root@formanserver ~]# systemctl stop pulpcore* --all
Warning: Stopping pulpcore-api.service, but it can still be activated by:
  pulpcore-api.socket
Warning: Stopping pulpcore-content.service, but it can still be activated by:
  pulpcore-content.socket
[root@formanserver ~]# cd /tmp
[root@formanserver tmp]# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell -c "import pulpcore; pulpcore.app.models.ReservedResource.objects.all().delete()"
[root@formanserver tmp]#

But I still can see old pulp tasks in waiting status :

[root@formanserver tmp]# curl https://`hostname`/pulp/api/v3/tasks/?state=waiting --cert /etc/pki/katello/certs/pulp-client.crt  --key /etc/pki/katello/private/pulp-client.key | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  9973  100  9973    0     0  47243      0 --:--:-- --:--:-- --:--:-- 47265
{
  "count": 19,
  "next": null,
  "previous": null,
  "results": [
    {
      "pulp_href": "/pulp/api/v3/tasks/1d9b45a0-fc76-46a5-947d-200a8b08500d/",
      "pulp_created": "2021-05-13T17:07:34.166746Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_update",
      "logging_cid": "7aa37155749344eab6d51fc120b9678e",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/2971eb63-fdea-4ba3-9a38-bc68deede1e0/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/ae84cabb-1531-4192-9b48-be973bf9f3e2/",
      "pulp_created": "2021-05-12T18:53:35.937847Z",
      "state": "waiting",
      "name": "pulp_deb.app.tasks.publishing.publish",
      "logging_cid": "5a002c96598f45b7aea925a79fda9d9b",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/827815db-7f88-4d77-89e1-d351a2bce120/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/042def9d-550b-4eb0-af31-ddfb7c7fa893/",
      "pulp_created": "2021-05-12T18:53:35.599286Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_update",
      "logging_cid": "72c6e943d2d4461d8d25e24449998172",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/45a584ae-3e51-45ab-8e33-aa003f84a91e/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/43e6ac99-fedb-416a-acbc-5872bf7113fb/",
      "pulp_created": "2021-05-12T18:53:34.914631Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_update",
      "logging_cid": "c28d58d04a054293a382f00880e3260b",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/remotes/deb/apt/cbe41858-7231-4b76-b292-71b1730cbfa3/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/a8d6c790-2d18-41bd-b7d4-892266d34584/",
      "pulp_created": "2021-05-12T18:53:34.473959Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_update",
      "logging_cid": "679c6e77133c417fb9f5437447b654ee",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/827815db-7f88-4d77-89e1-d351a2bce120/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/1a6fd4b9-4c8e-4f79-8957-3406da0bdbb0/",
      "pulp_created": "2021-05-12T09:49:22.053688Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "cda05c77fc4d4a4fb17a05e1be8b0435",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/9291001d-ec28-4fb6-96ad-264611c64f69/",
      "pulp_created": "2021-05-12T09:49:22.017887Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "10d768d618ea4c4a96b0eb45c2218634",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/9e520a02-6819-4b6b-a919-d0a98ae08878/",
      "pulp_created": "2021-05-12T09:49:21.978039Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "115a3110ccb54ab3a86ed222daea3915",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/b79c5fef-3333-4afb-8249-b4ea5787daac/",
      "pulp_created": "2021-05-12T09:49:21.937124Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "703886c2909943a5adfa05a82795602a",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/63fbb419-3c1e-4c8f-8d10-b604baf00d1e/",
      "pulp_created": "2021-05-12T09:49:00.128477Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "3c48339155b044618dd6bda7d825c447",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/45a584ae-3e51-45ab-8e33-aa003f84a91e/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/c4e0d43b-176c-4a26-a386-75cf85cde6b6/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/926775aa-4f59-46bd-877e-5c30d580661e/",
      "pulp_created": "2021-05-12T09:49:00.047113Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "938a6f4729ab477eb1de1d172650291e",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/ea23faa4-eb56-4d1b-bdc2-d9779fb91051/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/3c71dbc0-936b-47fd-ae31-12ddcbaf9377/",
      "pulp_created": "2021-05-12T09:48:59.913419Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "d110524ce2e348cf9a878e9117bfd1a6",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/ccfbef95-5387-467d-99e5-420ced8dedbf/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/3bed5208-06f1-44a2-9b98-ab449f231a41/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/4472cad9-8390-46c7-a9ba-3dbe06689049/",
      "pulp_created": "2021-05-12T09:48:42.350770Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "c9d9b62b4ff340a1b805346656573ada",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/107b4468-f579-4e2a-8939-70251f98150f/",
      "pulp_created": "2021-05-12T09:48:42.318951Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "de888e671f7444d392395acb40a3bdc3",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/c256f1a8-3be8-4fd2-a191-051c2d186b9b/",
      "pulp_created": "2021-05-12T09:48:42.285439Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "770a6a7d48e74080aebeb66ae6591783",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/f665b048-c19b-455b-a90d-06dde3c3c71c/",
      "pulp_created": "2021-05-12T09:48:15.892560Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "cd75feb1de3341e48bd429408577a0b1",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/8a756286-c136-485c-a412-df48c43d47aa/",
      "pulp_created": "2021-05-12T09:48:15.858513Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "efcc734a843e4589983299a8257bbede",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/34b6bd43-ea9d-43ef-9332-d18dae566546/",
      "pulp_created": "2021-05-12T09:48:15.826673Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "39b5eff1df4741c1abdc65418c01f1ed",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/f5505640-03aa-44d5-9c73-b40a919d5102/",
      "pulp_created": "2021-05-12T09:48:15.717927Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "d04844a051ca49d1ac120e8d7d9f3e8e",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    }
  ]
}

Below output of systemctl status pulpcore* --all

[root@formanserver tmp]# systemctl status pulpcore* --all
● pulpcore-api.service - Pulp API Server
   Loaded: loaded (/etc/systemd/system/pulpcore-api.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-05-13 17:08:54 UTC; 7min ago
 Main PID: 15535 (gunicorn)
   Status: "Gunicorn arbiter booted"
    Tasks: 2
   Memory: 98.9M
   CGroup: /system.slice/pulpcore-api.service
           ├─15535 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application -w 1 --access-logfile -
           └─15545 /usr/bin/python3 /usr/bin/gunicorn pulpcore.app.wsgi:application -w 1 --access-logfile -

May 13 17:12:28 formanserver pulpcore-api[15535]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 67af19e5e2b94bf7ace3407750bbfe09
May 13 17:12:28 formanserver pulpcore-api[15535]: - - [13/May/2021:17:12:28 +0000] "GET /pulp/api/v3/tasks/1d9b45a0-fc76-46a5-947d-200a8b08500d/ HTTP/1.1" 200 506 "-" "OpenAPI-Generator/3.9.0/ruby"
May 13 17:12:43 formanserver pulpcore-api[15535]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 2101d269a80d4c78954d7516f0721ee9
May 13 17:12:43 formanserver pulpcore-api[15535]: - - [13/May/2021:17:12:43 +0000] "GET /pulp/api/v3/tasks/ac44548e-d53c-4e42-8b52-393b22e5650e/ HTTP/1.1" 200 507 "-" "OpenAPI-Generator/3.9.0/ruby"
May 13 17:12:43 formanserver pulpcore-api[15535]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 1af9a3883fcb428e9ce4eb426992a665
May 13 17:12:43 formanserver pulpcore-api[15535]: - - [13/May/2021:17:12:43 +0000] "GET /pulp/api/v3/tasks/eb258680-32af-48f2-aa20-edb5c410540c/ HTTP/1.1" 200 507 "-" "OpenAPI-Generator/3.9.0/ruby"
May 13 17:12:43 formanserver pulpcore-api[15535]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: f3e42f9547304878af5a69429192bc54
May 13 17:12:43 formanserver pulpcore-api[15535]: - - [13/May/2021:17:12:43 +0000] "GET /pulp/api/v3/tasks/6fcb3756-0b6d-4eeb-a268-c07acbca8c63/ HTTP/1.1" 200 507 "-" "OpenAPI-Generator/3.9.0/ruby"
May 13 17:12:44 formanserver pulpcore-api[15535]: pulp [None]: django_guid:INFO: Header `Correlation-ID` was not found in the incoming request. Generated new GUID: 0aeb4ceeb9ff490ba9a7db66f2be273a
May 13 17:12:44 formanserver pulpcore-api[15535]: - - [13/May/2021:17:12:44 +0000] "GET /pulp/api/v3/tasks/1d9b45a0-fc76-46a5-947d-200a8b08500d/ HTTP/1.1" 200 507 "-" "OpenAPI-Generator/3.9.0/ruby"

● pulpcore-worker@2.service - Pulp RQ Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-05-13 17:08:53 UTC; 7min ago
 Main PID: 15524 (rq)
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@2.service
           └─15524 /usr/bin/python3 /usr/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -c pulpcore.rqconfig --disable-job-desc-logging

May 13 17:08:53 formanserver systemd[1]: Started Pulp RQ Worker.
May 13 17:08:59 formanserver pulpcore-worker-2[15524]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker '15524@formanserver'.
May 13 17:08:59 formanserver pulpcore-worker-2[15524]: pulp [None]: rq.worker:INFO: Worker rq:worker:15524@formanserver: started, version 1.7.0
May 13 17:08:59 formanserver pulpcore-worker-2[15524]: pulp [None]: rq.worker:INFO: Subscribing to channel rq:pubsub:15524@formanserver
May 13 17:08:59 formanserver pulpcore-worker-2[15524]: pulp [None]: rq.worker:INFO: *** Listening on 15524@formanserver...
May 13 17:08:59 formanserver pulpcore-worker-2[15524]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: New worker '15524@formanserver' discovered
May 13 17:08:59 formanserver pulpcore-worker-2[15524]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 15524@formanserver

● pulpcore-content.service - Pulp Content App
   Loaded: loaded (/etc/systemd/system/pulpcore-content.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-05-13 17:08:54 UTC; 7min ago
 Main PID: 15525 (gunicorn)
   Status: "Gunicorn arbiter booted"
    Tasks: 10
   Memory: 737.5M
   CGroup: /system.slice/pulpcore-content.service
           ├─15525 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
           ├─15546 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
           ├─15550 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
           ├─15552 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
           ├─15553 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
           ├─15554 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
           ├─15555 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
           ├─15558 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
           ├─15559 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -
           └─15560 /usr/bin/python3 /usr/bin/gunicorn pulpcore.content:server --worker-class aiohttp.GunicornWebWorker -w 9 --access-logfile -

May 13 17:08:54 formanserver systemd[1]: Started Pulp Content App.
May 13 17:08:54 formanserver pulpcore-content[15525]: [2021-05-13 17:08:54 +0000] [15546] [INFO] Booting worker with pid: 15546
May 13 17:08:54 formanserver pulpcore-content[15525]: [2021-05-13 17:08:54 +0000] [15550] [INFO] Booting worker with pid: 15550
May 13 17:08:54 formanserver pulpcore-content[15525]: [2021-05-13 17:08:54 +0000] [15552] [INFO] Booting worker with pid: 15552
May 13 17:08:54 formanserver pulpcore-content[15525]: [2021-05-13 17:08:54 +0000] [15553] [INFO] Booting worker with pid: 15553
May 13 17:08:54 formanserver pulpcore-content[15525]: [2021-05-13 17:08:54 +0000] [15554] [INFO] Booting worker with pid: 15554
May 13 17:08:54 formanserver pulpcore-content[15525]: [2021-05-13 17:08:54 +0000] [15555] [INFO] Booting worker with pid: 15555
May 13 17:08:54 formanserver pulpcore-content[15525]: [2021-05-13 17:08:54 +0000] [15558] [INFO] Booting worker with pid: 15558
May 13 17:08:54 formanserver pulpcore-content[15525]: [2021-05-13 17:08:54 +0000] [15559] [INFO] Booting worker with pid: 15559
May 13 17:08:54 formanserver pulpcore-content[15525]: [2021-05-13 17:08:54 +0000] [15560] [INFO] Booting worker with pid: 15560

● pulpcore-worker@4.service - Pulp RQ Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-05-13 17:08:53 UTC; 7min ago
 Main PID: 15526 (rq)
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@4.service
           └─15526 /usr/bin/python3 /usr/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -c pulpcore.rqconfig --disable-job-desc-logging

May 13 17:08:53 formanserver systemd[1]: Started Pulp RQ Worker.
May 13 17:09:01 formanserver pulpcore-worker-4[15526]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker '15526@formanserver'.
May 13 17:09:01 formanserver pulpcore-worker-4[15526]: pulp [None]: rq.worker:INFO: Worker rq:worker:15526@formanserver: started, version 1.7.0
May 13 17:09:01 formanserver pulpcore-worker-4[15526]: pulp [None]: rq.worker:INFO: Subscribing to channel rq:pubsub:15526@formanserver
May 13 17:09:01 formanserver pulpcore-worker-4[15526]: pulp [None]: rq.worker:INFO: *** Listening on 15526@formanserver...
May 13 17:09:01 formanserver pulpcore-worker-4[15526]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: New worker '15526@formanserver' discovered
May 13 17:09:01 formanserver pulpcore-worker-4[15526]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 15526@formanserver

● pulpcore-resource-manager.service - Pulp Resource Manager
   Loaded: loaded (/etc/systemd/system/pulpcore-resource-manager.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-05-13 17:08:53 UTC; 7min ago
 Main PID: 15527 (rq)
    Tasks: 2
   Memory: 82.1M
   CGroup: /system.slice/pulpcore-resource-manager.service
           └─15527 /usr/bin/python3 /usr/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -n resource-manager -c pulpcore.rqconfig --disable-job-desc-logging

May 13 17:10:52 formanserver pulpcore-resource-manager[15527]: pulp [None]: rq.worker:INFO: resource-manager: 7f516739-7f61-4c0c-b334-fc6f67a720bf
May 13 17:10:52 formanserver pulpcore-resource-manager[15527]: pulp [2dd34126aa3b4e2e93acd67b4f1327db]: rq.worker:INFO: resource-manager: Job OK (7f516739-7f61-4c0c-b334-fc6f67a720bf)
May 13 17:10:52 formanserver pulpcore-resource-manager[15527]: pulp [None]: rq.worker:INFO: resource-manager: 6837129a-b0b6-4218-bc0e-2613cfccc1b9
May 13 17:10:52 formanserver pulpcore-resource-manager[15527]: pulp [014ce39dc13d468d952f17d8a896c467]: rq.worker:INFO: resource-manager: Job OK (6837129a-b0b6-4218-bc0e-2613cfccc1b9)
May 13 17:12:33 formanserver pulpcore-resource-manager[15527]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: Worker '8633@formanserver' has gone missing, removing from list of workers
May 13 17:12:33 formanserver pulpcore-resource-manager[15527]: pulp [None]: pulpcore.tasking.services.worker_watcher:ERROR: The worker named 8633@formanserver is missing. Canceling the tasks in its queue.
May 13 17:12:33 formanserver pulpcore-resource-manager[15527]: pulp [None]: pulpcore.tasking.util:INFO: Canceling task: 1d9b45a0-fc76-46a5-947d-200a8b08500d
May 13 17:12:33 formanserver pulpcore-resource-manager[15527]: pulp [None]: pulpcore.tasking.util:INFO: Canceling task: ac44548e-d53c-4e42-8b52-393b22e5650e
May 13 17:12:34 formanserver pulpcore-resource-manager[15527]: pulp [None]: pulpcore.tasking.util:INFO: Canceling task: eb258680-32af-48f2-aa20-edb5c410540c
May 13 17:12:34 formanserver pulpcore-resource-manager[15527]: pulp [None]: pulpcore.tasking.util:INFO: Canceling task: 6fcb3756-0b6d-4eeb-a268-c07acbca8c63

● pulpcore-worker@3.service - Pulp RQ Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-05-13 17:08:53 UTC; 7min ago
 Main PID: 15528 (rq)
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@3.service
           └─15528 /usr/bin/python3 /usr/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -c pulpcore.rqconfig --disable-job-desc-logging

May 13 17:08:53 formanserver systemd[1]: Started Pulp RQ Worker.
May 13 17:09:00 formanserver pulpcore-worker-3[15528]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker '15528@formanserver'.
May 13 17:09:00 formanserver pulpcore-worker-3[15528]: pulp [None]: rq.worker:INFO: Worker rq:worker:15528@formanserver: started, version 1.7.0
May 13 17:09:00 formanserver pulpcore-worker-3[15528]: pulp [None]: rq.worker:INFO: Subscribing to channel rq:pubsub:15528@formanserver
May 13 17:09:00 formanserver pulpcore-worker-3[15528]: pulp [None]: rq.worker:INFO: *** Listening on 15528@formanserver...
May 13 17:09:00 formanserver pulpcore-worker-3[15528]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: New worker '15528@formanserver' discovered
May 13 17:09:00 formanserver pulpcore-worker-3[15528]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 15528@formanserver

● pulpcore-worker@1.service - Pulp RQ Worker
   Loaded: loaded (/etc/systemd/system/pulpcore-worker@.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2021-05-13 17:08:53 UTC; 7min ago
 Main PID: 15529 (rq)
   CGroup: /system.slice/system-pulpcore\x2dworker.slice/pulpcore-worker@1.service
           └─15529 /usr/bin/python3 /usr/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -c pulpcore.rqconfig --disable-job-desc-logging

May 13 17:08:53 formanserver systemd[1]: Started Pulp RQ Worker.
May 13 17:08:58 formanserver pulpcore-worker-1[15529]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: Cleaning up shutdown worker '15529@formanserver'.
May 13 17:08:58 formanserver pulpcore-worker-1[15529]: pulp [None]: rq.worker:INFO: Worker rq:worker:15529@formanserver: started, version 1.7.0
May 13 17:08:58 formanserver pulpcore-worker-1[15529]: pulp [None]: rq.worker:INFO: Subscribing to channel rq:pubsub:15529@formanserver
May 13 17:08:58 formanserver pulpcore-worker-1[15529]: pulp [None]: rq.worker:INFO: *** Listening on 15529@formanserver...
May 13 17:08:58 formanserver pulpcore-worker-1[15529]: pulp [None]: pulpcore.tasking.services.worker_watcher:INFO: New worker '15529@formanserver' discovered
May 13 17:08:58 formanserver pulpcore-worker-1[15529]: pulp [None]: rq.worker:INFO: Cleaning registries for queue: 15529@formanserver

and here Pulp workers:

{
  "count": 21,
  "next": null,
  "previous": null,
  "results": [
    {
      "pulp_href": "/pulp/api/v3/workers/d8271182-4fca-49c1-848c-de6c05784a59/",
      "pulp_created": "2021-05-13T17:09:01.266486Z",
      "name": "15526@formanserver",
      "last_heartbeat": "2021-05-13T17:17:22.450156Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/94f68319-31c7-4e40-b888-c579e32c8e25/",
      "pulp_created": "2021-05-13T17:09:00.053095Z",
      "name": "15528@formanserver",
      "last_heartbeat": "2021-05-13T17:17:22.249684Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/7cba785b-7531-47d1-8c9b-c50e0ff3d3c9/",
      "pulp_created": "2021-05-13T17:08:59.933685Z",
      "name": "15524@formanserver",
      "last_heartbeat": "2021-05-13T17:17:22.150230Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/a7f79c21-c919-44df-a2ef-b4419c8928fe/",
      "pulp_created": "2021-05-13T17:08:58.865349Z",
      "name": "15529@formanserver",
      "last_heartbeat": "2021-05-13T17:17:20.647289Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/9f17f8b0-d442-40a6-8501-060ea06368d5/",
      "pulp_created": "2021-05-12T19:08:23.327781Z",
      "name": "8635@formanserver",
      "last_heartbeat": "2021-05-13T17:07:25.808826Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/26ac8319-e622-4e7e-83f3-680b3902eb27/",
      "pulp_created": "2021-05-12T19:08:23.289755Z",
      "name": "8632@formanserver",
      "last_heartbeat": "2021-05-13T17:07:25.818912Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/2971eb63-fdea-4ba3-9a38-bc68deede1e0/",
      "pulp_created": "2021-05-12T19:08:22.862526Z",
      "name": "8633@formanserver",
      "last_heartbeat": "2021-05-13T17:12:35.095642Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/341b23e0-30d3-4e34-9ab2-edae26b54982/",
      "pulp_created": "2021-05-12T19:08:22.333356Z",
      "name": "8634@formanserver",
      "last_heartbeat": "2021-05-13T17:07:25.786884Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/8989bba9-aeab-409a-baf0-ab0ac42d0340/",
      "pulp_created": "2021-05-12T10:15:28.605294Z",
      "name": "18431@formanserver",
      "last_heartbeat": "2021-05-12T19:13:24.527504Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/5273f427-20ce-4a89-a5ac-76cf6d0644f9/",
      "pulp_created": "2021-05-12T10:15:28.240457Z",
      "name": "18441@formanserver",
      "last_heartbeat": "2021-05-12T19:11:59.079129Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/4899e382-d255-45ba-8ee7-c33a8f9733eb/",
      "pulp_created": "2021-05-12T10:15:27.533365Z",
      "name": "18440@formanserver",
      "last_heartbeat": "2021-05-12T19:13:24.018947Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/1a990a41-af81-4d21-b8c6-ba163e1dff05/",
      "pulp_created": "2021-05-12T10:15:25.442660Z",
      "name": "18439@formanserver",
      "last_heartbeat": "2021-05-12T19:13:23.501735Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/c72bc34e-f372-4376-bd65-c9066690577f/",
      "pulp_created": "2021-05-12T09:44:08.915210Z",
      "name": "1713@formanserver",
      "last_heartbeat": "2021-05-12T10:20:17.923549Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/c5bd1542-6c93-41ee-99f5-c955ee8e563d/",
      "pulp_created": "2021-05-12T09:44:08.104734Z",
      "name": "1697@formanserver",
      "last_heartbeat": "2021-05-12T10:20:17.910476Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/e05038dd-1e54-4cc8-8676-f7b96af455d3/",
      "pulp_created": "2021-05-12T09:44:07.870187Z",
      "name": "1709@formanserver",
      "last_heartbeat": "2021-05-12T10:20:17.919404Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/2e8a38b0-f879-4e15-9bd0-d640c14e9fd2/",
      "pulp_created": "2021-05-12T09:44:07.422913Z",
      "name": "1715@formanserver",
      "last_heartbeat": "2021-05-12T10:20:17.915188Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/32cbe404-fd67-4e7d-a248-0382bac60c0c/",
      "pulp_created": "2021-05-10T19:48:14.375531Z",
      "name": "resource-manager",
      "last_heartbeat": "2021-05-13T17:17:35.974697Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "pulp_created": "2021-05-10T19:48:09.909268Z",
      "name": "6347@formanserver",
      "last_heartbeat": "2021-05-12T18:56:45.955889Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "pulp_created": "2021-05-10T19:48:09.629296Z",
      "name": "6214@formanserver",
      "last_heartbeat": "2021-05-12T18:56:45.960076Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/45a584ae-3e51-45ab-8e33-aa003f84a91e/",
      "pulp_created": "2021-05-10T19:48:08.906922Z",
      "name": "6086@formanserver",
      "last_heartbeat": "2021-05-12T18:56:45.964094Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/ccfbef95-5387-467d-99e5-420ced8dedbf/",
      "pulp_created": "2021-05-10T19:48:08.191210Z",
      "name": "5955@formanserver",
      "last_heartbeat": "2021-05-12T18:49:12.252583Z"
    }
  ]
}

iballou · May 13, 2021, 6:27pm

@laimsi when you do a systemctl restart pulpcore* --all and then look at the number of online workers from the status endpoint, how many do you see? Since you have 4 pulpcore-worker services plus one resource manager, I would expect only 5 pulp workers in the status endpoint.

If there are more than 5, can you run ps -p <pid> on all of the pulp workers from the status endpoint? The pid is the number in front of the @ sign in the worker’s name. I’d like to know if all of those processes are actually running.

I have a feeling some of those workers aren’t being cleaned up because the last_heartbeat is a day old for some.

laimsi · May 13, 2021, 6:52pm

Hello @iballou

Only 5 PID show running process : 4 workers + resource manager

[root@foremanserver tmp]# for i in $(cat pid); do ps $i; done
10019 ?        Ssl    0:03 /usr/bin/python3 /usr/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -c pulpcore.rqconfig --disable-job-desc-logging
10010 ?        Ssl    0:04 /usr/bin/python3 /usr/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -c pulpcore.rqconfig --disable-job-desc-logging
10027 ?        Ssl    0:04 /usr/bin/python3 /usr/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -c pulpcore.rqconfig --disable-job-desc-logging
10017 ?        Ssl    0:04 /usr/bin/python3 /usr/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -c pulpcore.rqconfig --disable-job-desc-logging
10012 ?        Ssl    0:05 /usr/bin/python3 /usr/bin/rq worker -w pulpcore.tasking.worker.PulpWorker -n resource-manager -c pulpcore.rqconfig --disable-job-desc-logging

I have the same result using pulpcore-manager shell

[root@foremanserver tmp]# sudo -u pulp PULP_SETTINGS='/etc/pulp/settings.py' DJANGO_SETTINGS_MODULE='pulpcore.app.settings' pulpcore-manager shell <<EOF
from pulpcore.app.models import Worker
workers = [w.name for w in Worker.objects.online_workers()]
for rwork in workers:
  if rwork in workers:
    print(f'Worker {rwork}')
EOF
Worker 10017@foremanserver 
Worker 10027@foremanserver 
Worker 10010@foremanserver 
Worker 10019@foremanserver 
Worker resource-manager

Regards,

iballou · May 13, 2021, 7:14pm

That looks fine, but do you still see the extra workers when you query the status endpoint after the restart? Or do you only see those 5 that you printed above?

I’m guessing you only have the 5 since your ps command above only had 5 in it.

laimsi · May 13, 2021, 7:33pm

Hello @iballou

Looks like pulp endpoint status still show all workers history even after the restart, so my guess is that here we have all worker (dead + running workers)

[root@foremanserver ~]# curl https://`hostname`/pulp/api/v3/workers/  --cert /etc/pki/katello/certs/pulp-client.crt  --key /etc/pki/katello/private/pulp-client.key  | jq
{
  "count": 37,
  "next": null,
  "previous": null,
  "results": [
    {
      "pulp_href": "/pulp/api/v3/workers/a550b928-00dd-4120-ad13-3684fd7584ff/",
      "pulp_created": "2021-05-13T19:03:58.809276Z",
      "name": "1668@foremanserver",
      "last_heartbeat": "2021-05-13T19:25:35.835841Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/60aa23a9-fc3a-4313-b02e-5b1d3f1e98a2/",
      "pulp_created": "2021-05-13T19:03:58.743991Z",
      "name": "1654@foremanserver",
      "last_heartbeat": "2021-05-13T19:25:22.762238Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/9374fa13-9dcc-46cf-9afe-2bbff8f0c7ec/",
      "pulp_created": "2021-05-13T19:03:58.100518Z",
      "name": "1653@foremanserver",
      "last_heartbeat": "2021-05-13T19:25:22.910338Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/a82636f5-bb7b-4456-aee7-17231ca37e71/",
      "pulp_created": "2021-05-13T19:03:56.555437Z",
      "name": "1658@foremanserver",
      "last_heartbeat": "2021-05-13T19:25:22.846506Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/9102e1f4-c59a-47ea-ae84-918fc0f45304/",
      "pulp_created": "2021-05-13T18:08:14.705376Z",
      "name": "10019@foremanserver",
      "last_heartbeat": "2021-05-13T19:01:41.480059Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/0f518d29-3f8b-4f0c-a81b-28af2f6b4c62/",
      "pulp_created": "2021-05-13T18:08:14.179399Z",
      "name": "10010@foremanserver",
      "last_heartbeat": "2021-05-13T19:01:40.880128Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/d3199e5f-ab91-40d1-a0f2-5b09aecbc7d2/",
      "pulp_created": "2021-05-13T18:08:14.153266Z",
      "name": "10027@foremanserver",
      "last_heartbeat": "2021-05-13T19:01:40.779578Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/d6aad0e4-a522-4620-899f-acf297bca673/",
      "pulp_created": "2021-05-13T18:08:13.651049Z",
      "name": "10017@foremanserver",
      "last_heartbeat": "2021-05-13T19:01:40.678951Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/90cbdcdb-327a-488d-9c4b-0c808dc43065/",
      "pulp_created": "2021-05-13T17:31:42.428137Z",
      "name": "26701@foremanserver",
      "last_heartbeat": "2021-05-13T18:34:23.515731Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/91411f1a-03be-403b-b3f5-c4edb0d0cd1a/",
      "pulp_created": "2021-05-13T17:31:42.373061Z",
      "name": "26677@foremanserver",
      "last_heartbeat": "2021-05-13T18:07:17.443641Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/fc8b9d31-ba0b-4053-9a9a-116cc6c8e1ca/",
      "pulp_created": "2021-05-13T17:31:42.269198Z",
      "name": "26700@foremanserver",
      "last_heartbeat": "2021-05-13T18:07:17.439743Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/fc664998-785e-4800-ad78-423ac6ce6c85/",
      "pulp_created": "2021-05-13T17:31:40.980137Z",
      "name": "26702@foremanserver",
      "last_heartbeat": "2021-05-13T18:07:17.437649Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/9fb8274a-2c9b-4342-98c6-00e6eaa6fad3/",
      "pulp_created": "2021-05-13T17:30:40.044771Z",
      "name": "25070@foremanserver",
      "last_heartbeat": "2021-05-13T17:36:29.689844Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/e72de69d-ee72-4a71-ac9f-205a9a2995e7/",
      "pulp_created": "2021-05-13T17:30:39.539500Z",
      "name": "25063@foremanserver",
      "last_heartbeat": "2021-05-13T17:36:29.685318Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/2c052d0e-374f-4a72-a747-32cd4d2dd900/",
      "pulp_created": "2021-05-13T17:30:39.273674Z",
      "name": "25068@foremanserver",
      "last_heartbeat": "2021-05-13T17:36:29.680712Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/9cd24ff1-f9e9-4f3a-a3b8-5ceaeef5b8e1/",
      "pulp_created": "2021-05-13T17:30:38.717770Z",
      "name": "25049@foremanserver",
      "last_heartbeat": "2021-05-13T17:48:52.055121Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/d8271182-4fca-49c1-848c-de6c05784a59/",
      "pulp_created": "2021-05-13T17:09:01.266486Z",
      "name": "15526@foremanserver",
      "last_heartbeat": "2021-05-13T17:30:31.345292Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/94f68319-31c7-4e40-b888-c579e32c8e25/",
      "pulp_created": "2021-05-13T17:09:00.053095Z",
      "name": "15528@foremanserver",
      "last_heartbeat": "2021-05-13T17:30:31.346726Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/7cba785b-7531-47d1-8c9b-c50e0ff3d3c9/",
      "pulp_created": "2021-05-13T17:08:59.933685Z",
      "name": "15524@foremanserver",
      "last_heartbeat": "2021-05-13T17:30:31.339498Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/a7f79c21-c919-44df-a2ef-b4419c8928fe/",
      "pulp_created": "2021-05-13T17:08:58.865349Z",
      "name": "15529@foremanserver",
      "last_heartbeat": "2021-05-13T17:30:31.342916Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/9f17f8b0-d442-40a6-8501-060ea06368d5/",
      "pulp_created": "2021-05-12T19:08:23.327781Z",
      "name": "8635@foremanserver",
      "last_heartbeat": "2021-05-13T17:07:25.808826Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/26ac8319-e622-4e7e-83f3-680b3902eb27/",
      "pulp_created": "2021-05-12T19:08:23.289755Z",
      "name": "8632@foremanserver",
      "last_heartbeat": "2021-05-13T17:07:25.818912Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/2971eb63-fdea-4ba3-9a38-bc68deede1e0/",
      "pulp_created": "2021-05-12T19:08:22.862526Z",
      "name": "8633@foremanserver",
      "last_heartbeat": "2021-05-13T17:12:35.095642Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/341b23e0-30d3-4e34-9ab2-edae26b54982/",
      "pulp_created": "2021-05-12T19:08:22.333356Z",
      "name": "8634@foremanserver",
      "last_heartbeat": "2021-05-13T17:07:25.786884Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/8989bba9-aeab-409a-baf0-ab0ac42d0340/",
      "pulp_created": "2021-05-12T10:15:28.605294Z",
      "name": "18431@foremanserver",
      "last_heartbeat": "2021-05-12T19:13:24.527504Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/5273f427-20ce-4a89-a5ac-76cf6d0644f9/",
      "pulp_created": "2021-05-12T10:15:28.240457Z",
      "name": "18441@foremanserver",
      "last_heartbeat": "2021-05-12T19:11:59.079129Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/4899e382-d255-45ba-8ee7-c33a8f9733eb/",
      "pulp_created": "2021-05-12T10:15:27.533365Z",
      "name": "18440@foremanserver",
      "last_heartbeat": "2021-05-12T19:13:24.018947Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/1a990a41-af81-4d21-b8c6-ba163e1dff05/",
      "pulp_created": "2021-05-12T10:15:25.442660Z",
      "name": "18439@foremanserver",
      "last_heartbeat": "2021-05-12T19:13:23.501735Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/c72bc34e-f372-4376-bd65-c9066690577f/",
      "pulp_created": "2021-05-12T09:44:08.915210Z",
      "name": "1713@foremanserver",
      "last_heartbeat": "2021-05-12T10:20:17.923549Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/c5bd1542-6c93-41ee-99f5-c955ee8e563d/",
      "pulp_created": "2021-05-12T09:44:08.104734Z",
      "name": "1697@foremanserver",
      "last_heartbeat": "2021-05-12T10:20:17.910476Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/e05038dd-1e54-4cc8-8676-f7b96af455d3/",
      "pulp_created": "2021-05-12T09:44:07.870187Z",
      "name": "1709@foremanserver",
      "last_heartbeat": "2021-05-12T10:20:17.919404Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/2e8a38b0-f879-4e15-9bd0-d640c14e9fd2/",
      "pulp_created": "2021-05-12T09:44:07.422913Z",
      "name": "1715@foremanserver",
      "last_heartbeat": "2021-05-12T10:20:17.915188Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/32cbe404-fd67-4e7d-a248-0382bac60c0c/",
      "pulp_created": "2021-05-10T19:48:14.375531Z",
      "name": "resource-manager",
      "last_heartbeat": "2021-05-13T19:25:22.775829Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "pulp_created": "2021-05-10T19:48:09.909268Z",
      "name": "6347@foremanserver",
      "last_heartbeat": "2021-05-12T18:56:45.955889Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "pulp_created": "2021-05-10T19:48:09.629296Z",
      "name": "6214@foremanserver",
      "last_heartbeat": "2021-05-12T18:56:45.960076Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/45a584ae-3e51-45ab-8e33-aa003f84a91e/",
      "pulp_created": "2021-05-10T19:48:08.906922Z",
      "name": "6086@foremanserver",
      "last_heartbeat": "2021-05-12T18:56:45.964094Z"
    },
    {
      "pulp_href": "/pulp/api/v3/workers/ccfbef95-5387-467d-99e5-420ced8dedbf/",
      "pulp_created": "2021-05-10T19:48:08.191210Z",
      "name": "5955@foremanserver",
      "last_heartbeat": "2021-05-12T18:49:12.252583Z"
    }
  ]
}

laimsi · May 13, 2021, 7:47pm

Hello @iballou ,

So the 10000$ question is how can we delete/cleanup these zombi pulp tasks:

[root@formanserver~]# curl https://`hostname`/pulp/api/v3/tasks/?state=waiting --cert /etc/pki/katello/certs/pulp-client.crt  --key /etc/pki/katello/private/pulp-client.key | jq
{
  "count": 25,
  "next": null,
  "previous": null,
  "results": [
    {
      "pulp_href": "/pulp/api/v3/tasks/aedc074b-154b-4ade-870b-16fd20798b77/",
      "pulp_created": "2021-05-13T18:10:04.406312Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_create",
      "logging_cid": "c764efa507e6437cbaaa63af0e97b0c4",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/90cbdcdb-327a-488d-9c4b-0c808dc43065/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/75c597d0-a855-462c-921e-130296e3e723/",
      "pulp_created": "2021-05-13T17:42:12.810332Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_create",
      "logging_cid": "e363212c9e624c998a1d53e561c73e19",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9cd24ff1-f9e9-4f3a-a3b8-5ceaeef5b8e1/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/81ba10d3-ab4b-4555-8155-bd7f306d62e2/",
      "pulp_created": "2021-05-13T17:42:12.686424Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_create",
      "logging_cid": "f880b8010da94742a019ca0730ff5eff",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9cd24ff1-f9e9-4f3a-a3b8-5ceaeef5b8e1/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/eb568fdf-725b-4ba3-b1a7-5c67746adcc3/",
      "pulp_created": "2021-05-13T17:42:12.640478Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_create",
      "logging_cid": "ffce54a8b4374ca081d3e6f0b11adc70",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9cd24ff1-f9e9-4f3a-a3b8-5ceaeef5b8e1/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/303e258e-537b-4f81-8d17-574461c2c03b/",
      "pulp_created": "2021-05-13T17:38:36.842542Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "d6177b7999234a4a9ea8f7e1fe17ccbd",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9cd24ff1-f9e9-4f3a-a3b8-5ceaeef5b8e1/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/f3f01f33-aa4a-4d9e-9791-715113c8a79a/",
      "pulp_created": "2021-05-13T17:38:36.806879Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "2a1126604ca140469e8c82c25018844f",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9cd24ff1-f9e9-4f3a-a3b8-5ceaeef5b8e1/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/bb985d13-c1fc-4f37-ad1f-36420c46c05a/",
      "pulp_created": "2021-05-13T17:38:36.766661Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "6350213afc8a465a924725bab133e510",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9cd24ff1-f9e9-4f3a-a3b8-5ceaeef5b8e1/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/ae84cabb-1531-4192-9b48-be973bf9f3e2/",
      "pulp_created": "2021-05-12T18:53:35.937847Z",
      "state": "waiting",
      "name": "pulp_deb.app.tasks.publishing.publish",
      "logging_cid": "5a002c96598f45b7aea925a79fda9d9b",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/827815db-7f88-4d77-89e1-d351a2bce120/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/042def9d-550b-4eb0-af31-ddfb7c7fa893/",
      "pulp_created": "2021-05-12T18:53:35.599286Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_update",
      "logging_cid": "72c6e943d2d4461d8d25e24449998172",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/45a584ae-3e51-45ab-8e33-aa003f84a91e/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/43e6ac99-fedb-416a-acbc-5872bf7113fb/",
      "pulp_created": "2021-05-12T18:53:34.914631Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_update",
      "logging_cid": "c28d58d04a054293a382f00880e3260b",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/remotes/deb/apt/cbe41858-7231-4b76-b292-71b1730cbfa3/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/a8d6c790-2d18-41bd-b7d4-892266d34584/",
      "pulp_created": "2021-05-12T18:53:34.473959Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_update",
      "logging_cid": "679c6e77133c417fb9f5437447b654ee",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/827815db-7f88-4d77-89e1-d351a2bce120/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/1a6fd4b9-4c8e-4f79-8957-3406da0bdbb0/",
      "pulp_created": "2021-05-12T09:49:22.053688Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "cda05c77fc4d4a4fb17a05e1be8b0435",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/9291001d-ec28-4fb6-96ad-264611c64f69/",
      "pulp_created": "2021-05-12T09:49:22.017887Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "10d768d618ea4c4a96b0eb45c2218634",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/9e520a02-6819-4b6b-a919-d0a98ae08878/",
      "pulp_created": "2021-05-12T09:49:21.978039Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "115a3110ccb54ab3a86ed222daea3915",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/b79c5fef-3333-4afb-8249-b4ea5787daac/",
      "pulp_created": "2021-05-12T09:49:21.937124Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "703886c2909943a5adfa05a82795602a",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/63fbb419-3c1e-4c8f-8d10-b604baf00d1e/",
      "pulp_created": "2021-05-12T09:49:00.128477Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "3c48339155b044618dd6bda7d825c447",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/45a584ae-3e51-45ab-8e33-aa003f84a91e/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/c4e0d43b-176c-4a26-a386-75cf85cde6b6/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/926775aa-4f59-46bd-877e-5c30d580661e/",
      "pulp_created": "2021-05-12T09:49:00.047113Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "938a6f4729ab477eb1de1d172650291e",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/9672bf48-032d-4625-a377-1cd0c7fe36f8/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/ea23faa4-eb56-4d1b-bdc2-d9779fb91051/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/3c71dbc0-936b-47fd-ae31-12ddcbaf9377/",
      "pulp_created": "2021-05-12T09:48:59.913419Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "d110524ce2e348cf9a878e9117bfd1a6",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/ccfbef95-5387-467d-99e5-420ced8dedbf/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/pulp/api/v3/repositories/deb/apt/3bed5208-06f1-44a2-9b98-ab449f231a41/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/4472cad9-8390-46c7-a9ba-3dbe06689049/",
      "pulp_created": "2021-05-12T09:48:42.350770Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "c9d9b62b4ff340a1b805346656573ada",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/107b4468-f579-4e2a-8939-70251f98150f/",
      "pulp_created": "2021-05-12T09:48:42.318951Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "de888e671f7444d392395acb40a3bdc3",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/c256f1a8-3be8-4fd2-a191-051c2d186b9b/",
      "pulp_created": "2021-05-12T09:48:42.285439Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "770a6a7d48e74080aebeb66ae6591783",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/f665b048-c19b-455b-a90d-06dde3c3c71c/",
      "pulp_created": "2021-05-12T09:48:15.892560Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "cd75feb1de3341e48bd429408577a0b1",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/8a756286-c136-485c-a412-df48c43d47aa/",
      "pulp_created": "2021-05-12T09:48:15.858513Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "efcc734a843e4589983299a8257bbede",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/34b6bd43-ea9d-43ef-9332-d18dae566546/",
      "pulp_created": "2021-05-12T09:48:15.826673Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "39b5eff1df4741c1abdc65418c01f1ed",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    },
    {
      "pulp_href": "/pulp/api/v3/tasks/f5505640-03aa-44d5-9c73-b40a919d5102/",
      "pulp_created": "2021-05-12T09:48:15.717927Z",
      "state": "waiting",
      "name": "pulpcore.app.tasks.base.general_delete",
      "logging_cid": "d04844a051ca49d1ac120e8d7d9f3e8e",
      "started_at": null,
      "finished_at": null,
      "error": null,
      "worker": "/pulp/api/v3/workers/7caaee7d-c496-40b7-9069-0e063c2440a5/",
      "parent_task": null,
      "child_tasks": [],
      "task_group": null,
      "progress_reports": [],
      "created_resources": [],
      "reserved_resources_record": [
        "/api/v3/distributions/"
      ]
    }
  ]
}

iballou · May 13, 2021, 7:51pm

You pasted the /workers endpoint, does curl https://foremanserver/pulp/api/v3/status/ show the dead workers too?

Also, do you see any messages similar to the following in /var/log/messages?

May 13 19:39:10 server.example.com rq[46406]: pulp [None]: pulpcore.tasking.worker_watcher:ERROR: The worker named 46249@server.example.com is missing. Canceling the tasks in its queue.

I have another idea to try. We can manually cancel all of the stuck tasks in Pulp 3. To make this easier, I’d like to have you set up the pulp-cli:

pip install pulp-cli
Open up ~/.config/pulp/settings.toml and add the following:

[cli]
base_url = "https://<your FQDN>"
cert = "/etc/pki/katello/certs/pulp-client.crt"
key = "/etc/pki/katello/private/pulp-client.key"
verify_ssl = false

Then, we can cancel the tasks:

Find the tasks with pulp task list --state waiting
Cancel the tasks like so: pulp task cancel --href <task pulp_href>

I’m not certain if that will kill the waiting tasks, but it should in theory. Let me know if it doesn’t.