"Actions Pulp Repository RegenerateApplicability" stuck suspended, can't restart, can't cancel

Support
1

Problem:

There are two issues here.

  1. Actions::Pulp::Repository::RegenerateApplicability has stopped and it needs to be restarted/cancelled
  2. Cancel buttons (various) don’t cancel pulp action

17: Actions::Pulp::Repository::RegenerateApplicability (suspended) [ 30425.65s / 236.99s ] Cancel

Started at: 2018-05-16 16:07:25 UTC

Ended at: 2018-05-17 00:34:31 UTC

Real time: 30425.65s

Execution time (excluding suspended state): 236.99s

Input:

pulp_id: ORG-CentOS_7_x86_64-Updates
contents_changed: false
remote_user: admin
remote_cp_user: admin

Output:

pulp_task_group:
group_id: d1678585-e716-4a72-9b3b-f7389d74edc7
accepted: 0
finished: 468
running: 0
canceled: 0
waiting: 2
skipped: 0
suspended: 0
error: 0
total: 470
poll_attempts:
total: 1880
failed: 0

Solutions listed here

Expected outcome:

  1. Actions don’t enter this state if it’s a non completion state and it can’t be cancelled.

Foreman and Proxy versions:

1.17.0-1

Foreman and Proxy plugin versions:

katello-3.6.0-3.el7.noarch
pulp-admin-client-2.15.3-1.el7.noarch
pulp-client-1.0-1.noarch
pulp-deb-plugins-1.6.0-1.el7.noarch
pulp-docker-plugins-3.1.2-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch
pulp-puppet-plugins-2.15.3-1.el7.noarch
pulp-puppet-tools-2.15.3-1.el7.noarch
pulp-rpm-admin-extensions-2.15.3-1.el7.noarch
pulp-rpm-handlers-2.15.3-1.el7.noarch
pulp-rpm-plugins-2.15.3-1.el7.noarch
pulp-selinux-2.15.3-1.el7.noarch
pulp-server-2.15.3-1.el7.noarch

2

Sorry. The sentence “Solutions listed here” is unfinished. Missing the words “do not work”. I tried them without success.

3

Hello,
when the regenerate applicability gets triggered, it kicks off the task in pulp and suspends (basically goes to sleep). Then it wakes up regularly and polls pulp for the state of the task in pulp. If the task in pulp is not done yet, the dynflow task suspends itself. This repeats until the task in pulp is done.

Cancelling the task sends a request to pulp to cancel the task there and we rely on the suspend-poll mechanism to detect when the task in pulp actually cancelled itself.

Does the task in dynflow keep waking itself up (does the Execution time change)? What is the status of the task in pulp?

# I'm a bit rusty with pulp-admin so a bit of tweaking might be needed
$ pulp-admin tasks details --task-id d1678585-e716-4a72-9b3b-f7389d74edc7
1 Like
4

One thing i look at is the finished and waiting numbers. As long as waiting goes down and finished is going up it’s working. Suspended is a little misleading. I’ve had some repo syncs take a very long time and other go very quickly though this part. Now I’ve seen other issues cause this to ‘get stuck’ such as the pulp filesystem being filled up. I’ve had this happen when I’ve been on vacation and I get a call because “Foreman isn’t working”. There is a Red Hat solution that I’ve found works to clean things up. Then you fix whatever the issue is and restart the sync. I haven’t had a chance to try this in 1.17 / 3.6 only 1.16 / 3.5. Hopefully this script can help. Only use it as a last resort though.

#!/bin/bash

systemctl stop foreman-tasks

echo -e "Copy and paste the following command into the foreman-rake console:\n\nForemanTasks::Task.where(:state => :paused).destroy_all\n\n"

foreman-rake console

systemctl start foreman-tasks

STATE=""
for TASK in `pulp-admin tasks list | egrep '^Task Id:|^State:' | sed -e 's,^Task Id: ,,' -e 's,^State: ,,'`; do
    if [ "$STATE" = "" ]; then
            STATE=$TASK
    else
            if [ $STATE != Successful ] && [ $STATE != Cancelled ] && [ $STATE != Failed ]; then
                    pulp-admin tasks details --task-id=$TASK
                    pulp-admin tasks cancel --task-id=$TASK
            fi
            STATE=""
    fi
done
1 Like
5

Everything seems to have fixed itself overnight, so that’s happy making. I’m almost certain that’s not always the case (this has happened to me before).

But thank you both for new understanding and tools to test what’s going on.

1 Like