Foreman and Proxy plugin versions: katello 4.11, foreman-tasks 9.0.1
Distribution and version: Rocky 8
Other relevant data:
Since a few days ago none of katello related tasks are working.
This is a all-in-one installation with database internally running.
As I was deleting a content-view version I noticed that it’s stuck. I cancelled and cleaned up the job using foreman-rake foreman_tasks:cleanup TASK_SEARCH='result == pending' STATES='running' and other variations of warning, pending and error and restarted foreman afterwards. it still didn’t work after retrying.
After that I tried syncing repositories, but they don’t work either. on dynflow console I see this similar message where a task is waiting with waiting for pulp to start the task $something where “something” is different based on what I’m trying to do.
I Have searched the forum but couldn’t find anything relevant as most topics are old and troubleshooting methods in there don’t work. any kind of help is appreciated.
Not sure what is the cause but I think everything is running extremely slow. I have 8 pulpcore worker services running.
a few hours ago finally my repository sync job completed after 3 hours. after that I tried to sync another repo and it worked fine. removal of the mentioned content view also finished pretty quickly.
Right now I’m trying to publish a new content view. This usually takes about 10-15 minutes, but it has been running for more than an hour now. on dynflow-console it’s Actions::Pulp3::Repository::MultiCopyContent (waiting for Pulp to finish the task copy_content (ID: e7d253e521fc)) that has been running for a very very long time.
I still don’t know what’s going on. I managed to upgrade Foreman to 3.10& Katello 4.12. But the issue is still here. There are no running tasks.
content view publishing hangs, removing a version hangs, and like mentioned above, everything is waiting for pulpcore to start the task. have rebooted the server and as mentioned I had a successful upgrade, but it’s still the same.
Anyone has any idea how I should proceed?
I can see a task (or series of tasks) running, which is from a cancelled content-view publish. I can see the task in the running state when I run pulp task list --state running, but I can’t cancel it using pulp CLI as I get the message "Error: Call aborted due to safe mode". This seems like to be intentional as pulp CLI is read only as I understand, and because of Foreman, this task needs to be cancelled via Foreman UI or hammer CLI.
The problem is this task/execution plan, doesn’t show up on foreman UI or dynflow console.
I need a way to clean out all pulp tasks, workers etc. but still not any chance.
Hah, sure. The cli is configured for foreman to be readonly.
Add --force to the command and it should do it anyway.
Alternatively you can remove the dry-run option in the config. But yeah, in the forman installation we put it there for a reason.
Yeah, Managed to delete the tasks using the pulp CLI, and after that, a test repo sync worked without any issue.
But content view publish still wasn’t working still stuck at that same step. After further inspection and making sure no other task is running, it looked to me like it wasn’t stuck, but rather it was super slow probably. not that there was any progress in the overall progress bar.
and finally found the problem:
apparently someone had applied a filter to the whole content-view, filtering all *.RPMs. after removing it everything started to work properly like before.
