Problem:
Since I updated our dev environment from 1.22/Katello 3.12 to 2.0/3.15 this week, Foreman-Tasks Cleanup rake job seems borked.
After the upgrade, I encountered some broken “Pulp Disk Space Notification Tasks” beeing stuck at “running/100%/warning”. Since all of them were during the maintenance window (which had some trouble due to self-registred Katello beeing a pain to upgrade sometimes), and the error message complained about not beeing able to connect to Foreman Instance, I assumed they were planned tasks broken during the upgrade window.
Since those tasks can not be cancled, I went on and tried to get rid of them via foreman-rake task cleanup as described by the tasks plugin.
This resulted in the following stacktrace:
# foreman-rake foreman_tasks:cleanup TASK_SEARCH='label=CreatePulpDiskSpaceNotifications' STATES=running --trace
Rubocop not loaded.
Rubocop not loaded.
** Invoke foreman_tasks:cleanup (first_time)
** Invoke foreman_tasks:cleanup:run (first_time)
** Invoke environment (first_time)
** Execute environment
** Execute foreman_tasks:cleanup:run
rake aborted!
The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.3/lib/dynflow/rails.rb:75:in `world'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/lib/foreman_tasks/cleaner.rb:80:in `initialize'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/lib/foreman_tasks/cleaner.rb:8:in `new'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/lib/foreman_tasks/cleaner.rb:8:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-1.1.1/lib/foreman_tasks/tasks/cleanup.rake:37:in `block (3 levels) in <top (required)>'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `block in execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in `execute'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:217:in `block in invoke_prerequisites'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:215:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:215:in `invoke_prerequisites'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:194:in `block in invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in `mon_synchronize'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in `invoke_with_call_chain'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in `invoke'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in `invoke_task'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `each'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in `block in top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in `run_with_threads'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in `top_level'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in `block in run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in `run'
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in `<top (required)>'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `load'
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in `<main>'
Tasks: TOP => foreman_tasks:cleanup => foreman_tasks:cleanup:run
I am unsure what Dynflow world would be uninitialized, though. Dynflow is up and running and all tasks since the upgrade seem to be working fine.
I also went ahead and checked/validated the Dynflow wolds via the “/worlds” page, but with no success on this matter.
Expected outcome:
Task cleanup should work as expected.
Foreman and Proxy versions:
2.0.1
Foreman and Proxy plugin versions:
foreman-tasks The goal of this plugin is to unify the way of showing task statuses across the Foreman instance. It defines Task model for keeping the information about the tasks and Lock for assigning the tasks to resources. The locking allows dealing with preventing multiple colliding tasks to be run on the same resource. It also optionally provides Dynflow infrastructure for using it for managing the tasks. Ivan Nečas 1.1.1
foreman_bootdisk Plugin for Foreman that creates iPXE-based boot disks to provision hosts without the need for PXE infrastructure. Dominic Cleal 16.0.0
foreman_discovery MaaS Discovery Plugin engine for Foreman Aditi Puntambekar, alongoldboim, Alon Goldboim, amirfefer, Amit Karsale, Amos Benari, Avi Sharvit, Bryan Kearney, bshuster, Daniel Lobato, Daniel Lobato Garcia, Daniel Lobato García, Danny Smit, David Davis, Djebran Lezzoum, Dominic Cleal, Eric D. Helms, Ewoud Kohl van Wijngaarden, Frank Wall, Greg Sutcliffe, ChairmanTubeAmp, Ido Kanner, imriz, Imri Zvik, Ivan Nečas, Joseph Mitchell Magen, June Zhang, kgaikwad, Lars Berntzon, ldjebran, Lukas Zapletal, Lukáš Zapletal, Marek Hulan, Marek Hulán, Martin Bačovský, Matt Jarvis, Michael Moll, Nick, odovzhenko, Ohad Levy, Ondrej Prazak, Ondřej Ezr, Ori Rabin, orrabin, Partha Aji, Petr Chalupa, Phirince Philip, Rahul Bajaj, Robert Antoni Buj Gelonch, Scubafloyd, Sean O\'Keeffe, Sebastian Gräßl, Shimon Shtein, Shlomi Zadok, Stephen Benjamin, Swapnil Abnave, Thomas Gelf, Timo Goebel, Tomas Strych, Tom Caspy, Tomer Brisker, and Yann Cézard 16.0.1
foreman_hooks Plugin engine for Foreman that enables running custom hook scripts on Foreman events Dominic Cleal 0.3.16
foreman_remote_execution A plugin bringing remote execution to the Foreman, completing the config management functionality with remote management functionality. Foreman Remote Execution team 3.2.2
foreman_scc_manager Foreman plugin to sync SUSE Customer Center products and repositories into Katello. ATIX AG 1.8.3
foreman_snapshot_management Foreman-plugin to manage snapshots in a virtual-hardware environments. ATIX AG 1.7.1
foreman_templates Engine to synchronise provisioning templates from GitHub Greg Sutcliffe 8.0.0
katello Katello adds Content and Subscription Management to Foreman. For this it relies on Candlepin and Pulp. N/A 3.15.2
Distribution and version:
RHEL 7
