Problem: “Update Content Counts” task not terminating under /foreman_tasks/tasks
Expected outcome: Task
Foreman and Proxy versions: 3.17.2
Distribution and version: CentOS Stream 9
Other relevant data:
After creating a content view version and promotion I noticed the “Update Content Counts” tasks were scheduled, ran to completion in Dynflow but not finishing under /foreman_tasks/tasks – they are shown in state=planned with result=pending for hours.
Getting the task id from the dynflow console I can see the following log from production.log:
2026-06-23T07:12:45 [I|bac|fad243c6] Task {label: , execution_plan_id: 2a3f34e6-2705-4d42-b995-c5cda8931d8c} state changed: pending
2026-06-23T07:12:45 [I|bac|fad243c6] Task {label: Actions::Katello::CapsuleContent::UpdateContentCounts, id: 1c562e7f-f33d-480e-87b1-99073cb04012, execution_plan_id: 2a3f34e6-2705-4d42-b995-c5cda8931d8c} state changed: planning
2026-06-23T07:12:45 [I|bac|fad243c6] Task {label: Actions::Katello::CapsuleContent::UpdateContentCounts, id: 1c562e7f-f33d-480e-87b1-99073cb04012, execution_plan_id: 2a3f34e6-2705-4d42-b995-c5cda8931d8c} state changed: planned
2026-06-23T07:12:45 [I|bac|fad243c6] Task {label: Actions::Katello::CapsuleContent::UpdateContentCounts, execution_plan_id: 2a3f34e6-2705-4d42-b995-c5cda8931d8c} state changed: running
2026-06-23T07:12:48 [I|bac|fad243c6] Task {label: Actions::Katello::CapsuleContent::UpdateContentCounts, execution_plan_id: 2a3f34e6-2705-4d42-b995-c5cda8931d8c} state changed: stopped result: success
Is there something wrong? Any docs other than this to rtfm on? Any help will be appreciated!
The good news is this should be mostly cosmetic problem. Tasks is not the authoritative source of truth, Dynflow is so as long as things are finishing in Dynflow, you’re relatively safe.
Any docs other than this to rtfm on?
Not really. The gist is roughly this: The synchronization from dynflow to tasks is implemented using execution plan hooks - one each state change, a callback is executed that updates the task record. Since you’re seeing those things in production.log, it seems this mechanism is firing. Now if anything went wrong during that, it should be logged to production.log, but it wouldn’t hurt to check the output of journalctl -u dynflow-sidekiq@orchestrator, just in case.
1 Like
Good to hear fro you @aruzicka !
Here are the journalctl outputs:
Jun 23 05:42:14 tfm.example.org systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jun 23 05:42:19 tfm.example.org dynflow-sidekiq@orchestrator[1314]: 2026-06-23T05:42:19.142Z pid=1314 tid=b6 INFO: Enabling systemd notification integration
Jun 23 05:42:29 tfm.example.org dynflow-sidekiq@orchestrator[1314]: 2026-06-23T05:42:29.554Z pid=1314 tid=b6 INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"}
Jun 23 05:42:29 tfm.example.org dynflow-sidekiq@orchestrator[1314]: 2026-06-23T05:42:29.555Z pid=1314 tid=b6 INFO: GitLab reliable fetch activated!
Jun 23 05:43:02 tfm.example.org systemd[1]: Started Foreman jobs daemon - orchestrator on sidekiq.
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: E, [2026-06-23T06:20:40.066899 #1314] ERROR -- /parallel-executor-core: no manager for Dynflow::Director::Event[request_id: a6733c01-44e2-4f15-be6e-efe073c86521-32, execution_plan_id: 417ca9fc-ace0-4647-b24a-14bfeacc281d, step_id: 3, event: Dynflow::Action::Cancellable::Cancel, result: #<Concurrent::Promises::ResolvableFuture:0x00007fc9b19bd610 pending>, optional: false] (Dynflow::Error)
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: /usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/director.rb:200:in `handle_event'
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: /usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/abstract/core.rb:36:in `handle_event'
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: [ concurrent-ruby ]
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: /usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/abstract/core.rb:127:in `on_message'
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: [ concurrent-ruby ]
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: E, [2026-06-23T06:20:40.160120 #1314] ERROR -- /parallel-executor-core: no manager for Dynflow::Director::Event[request_id: a6733c01-44e2-4f15-be6e-efe073c86521-33, execution_plan_id: 4444bc77-1928-40d4-8b82-ab8d3ae12472, step_id: 3, event: Dynflow::Action::Cancellable::Cancel, result: #<Concurrent::Promises::ResolvableFuture:0x00007fc9b19b3d40 pending>, optional: false] (Dynflow::Error)
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: /usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/director.rb:200:in `handle_event'
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: /usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/abstract/core.rb:36:in `handle_event'
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: [ concurrent-ruby ]
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: /usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/abstract/core.rb:127:in `on_message'
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: [ concurrent-ruby ]
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: E, [2026-06-23T06:20:40.230042 #1314] ERROR -- /parallel-executor-core: no manager for Dynflow::Director::Event[request_id: a6733c01-44e2-4f15-be6e-efe073c86521-34, execution_plan_id: 666937f6-08d5-450b-8c74-2f0c13381c8d, step_id: 3, event: Dynflow::Action::Cancellable::Cancel, result: #<Concurrent::Promises::ResolvableFuture:0x00007fc9b198cf60 pending>, optional: false] (Dynflow::Error)
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: /usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/director.rb:200:in `handle_event'
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: /usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/abstract/core.rb:36:in `handle_event'
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: [ concurrent-ruby ]
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: /usr/share/gems/gems/dynflow-1.9.3/lib/dynflow/executors/abstract/core.rb:127:in `on_message'
Jun 23 06:20:40 tfm.example.org dynflow-sidekiq@orchestrator[1314]: [ concurrent-ruby ]
Jun 23 06:27:37 tfm.example.org systemd[1]: Stopping Foreman jobs daemon - orchestrator on sidekiq...
Jun 23 06:27:39 tfm.example.org systemd[1]: dynflow-sidekiq@orchestrator.service: Deactivated successfully.
Jun 23 06:27:39 tfm.example.org systemd[1]: Stopped Foreman jobs daemon - orchestrator on sidekiq.
Jun 23 06:27:39 tfm.example.org systemd[1]: dynflow-sidekiq@orchestrator.service: Consumed 2min 12.377s CPU time.
Jun 23 06:35:24 tfm.example.org systemd[1]: Starting Foreman jobs daemon - orchestrator on sidekiq...
Jun 23 06:35:24 tfm.example.org dynflow-sidekiq@orchestrator[81580]: 2026-06-23T06:35:24.589Z pid=81580 tid=1q1o INFO: Enabling systemd notification integration
Jun 23 06:35:27 tfm.example.org dynflow-sidekiq@orchestrator[81580]: 2026-06-23T06:35:27.165Z pid=81580 tid=1q1o INFO: Booting Sidekiq 6.5.12 with Sidekiq::RedisConnection::RedisAdapter options {:url=>"redis://localhost:6379/6"}
Jun 23 06:35:27 tfm.example.org dynflow-sidekiq@orchestrator[81580]: 2026-06-23T06:35:27.167Z pid=81580 tid=1q1o INFO: GitLab reliable fetch activated!
Jun 23 06:35:37 tfm.example.org systemd[1]: Started Foreman jobs daemon - orchestrator on sidekiq.
Pretty sure the errors were the result of me clicking cancel from the webUI.
The restarts were due to me patching the OS prepping for upgrades.
The sync was done after the restart and it was error-free I think.
So now I have 3 “Update Content Counts” tasks from the UI which cannot be Cancel/Resume/Force Cancel. Should I clear that with the rake task? Like:
root@tfm:yum.repos.d$ foreman-rake foreman_tasks:cleanup TASK_SEARCH="*" STATES="planned" VERBOSE=true NOOP=true
[noop] deleting all tasks matching filter (*) AND (state ^ (planned))
[noop] 3 tasks matching filter would be deleted
[noop] 0 orphaned task locks would be deleted
[noop] 0 orphaned task links would be deleted
[noop] 0 orphaned execution plans would be deleted
[noop] 0 orphaned job invocations would be deleted
Went ahead and cleaned the tasks, now running 3.18.1 
Pretty much, but those are “safe” errors.
Yeah, that should do the trick, but please keep an eye out for it. If it happens again, we shouldn’t dismiss it just as a fluke.
1 Like
