Problem: dynflow-sidekiq@orchestrator service is not starting getting the below error in the messages log. Unsure why this happened or if any changes happened. it was noticed after a reboot
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: (): did not find expected key while parsing a block mapping at line 15 column 3
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby27/root/usr/share/ruby/psych.rb:456:in parse' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby27/root/usr/share/ruby/psych.rb:456:in parse_stream’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby27/root/usr/share/ruby/psych.rb:390:in parse' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby27/root/usr/share/ruby/psych.rb:277:in load’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /usr/share/foreman/config/settings.rb:34:in block in <top (required)>' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /usr/share/foreman/config/settings.rb:33:in each’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /usr/share/foreman/config/settings.rb:33:in <top (required)>' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in require’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in require' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in block in require’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in load_dependency' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in require’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /usr/share/foreman/config/application.rb:26:in <top (required)>' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in require’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in require' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /usr/share/foreman/extras/dynflow-sidekiq.rb:6:in <top (required)>’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in require' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in require’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:292:in boot_system' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/cli.rb:46:in run’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/bin/sidekiq:12:in <top (required)>' Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in load’
Aug 1 12:14:52 locahost dynflow-sidekiq@orchestrator: /opt/theforeman/tfm/root/usr/bin/sidekiq:23:in `’
Expected outcome:
Foreman and Proxy versions:
foreman-dynflow-sidekiq-3.0.2
foreman-proxy-3.0.2
foreman-3.0.2
katello-4.2.2
Foreman and Proxy plugin versions:
Distribution and version:
CentOS Linux 7 (Core)
Other relevant data:
Looks like i have a large number of failed jobs… any idea how i can delete these or clear them without foreman-rake
sudo su - postgres -c “psql -d foreman -c ‘select label,count(label),state,result from foreman_tasks_tasks where state <> ‘'‘stopped’'’ group by label,state,result ORDER BY label;’” label | count | state | result
--------------------------------------------------±------±----------±--------
Actions::Katello::Repository::Destroy | 1 | paused | error
Actions::Katello::Repository::Discover | 1 | paused | pending
Actions::Katello::Repository::MetadataGenerate | 1 | planning | pending
Actions::Katello::Repository::Sync | 5 | running | pending
Actions::Katello::SyncPlan::Run | 1 | running | pending
Actions::Katello::SyncPlan::Run | 7 | scheduled | pending
Actions::RemoteExecution::RunHostJob | 73 | paused | pending
Actions::RemoteExecution::RunHostJob | 1 | planning | pending
Actions::RemoteExecution::RunHostJob | 4401 | running | pending
Actions::RemoteExecution::RunHostsJob | 1 | paused | error
Actions::RemoteExecution::RunHostsJob | 3 | paused | pending
Actions::RemoteExecution::RunHostsJob | 76 | running | error
Actions::RemoteExecution::RunHostsJob | 14 | running | pending
Actions::RemoteExecution::RunHostsJob | 1 | scheduled | pending
Actions::SccManager::SyncPlanAccountRepositories | 3 | scheduled | pending
CreateExpiredManifestNotifications | 1 | scheduled | pending
CreatePulpDiskSpaceNotifications | 1 | scheduled | pending
CreateRssNotifications | 1 | scheduled | pending
SendExpireSoonNotifications | 1 | scheduled | pending
StoredValuesCleanupJob | 1 | scheduled | pending
| 0 | pending | success
(21 rows)
foreman-rake foreman_tasks:cleanup TASK_SEARCH=‘label = Actions::RemoteExecution::RunHostJob’ VERBOSE=‘VERBOSE=true’ --trace
rake aborted!
Psych::SyntaxError: (): did not find expected key while parsing a block mapping at line 15 column 3
/opt/rh/rh-ruby27/root/usr/share/ruby/psych.rb:456:in parse' /opt/rh/rh-ruby27/root/usr/share/ruby/psych.rb:456:in parse_stream’
/opt/rh/rh-ruby27/root/usr/share/ruby/psych.rb:390:in parse' /opt/rh/rh-ruby27/root/usr/share/ruby/psych.rb:277:in load’
/usr/share/foreman/config/settings.rb:34:in block in <top (required)>' /usr/share/foreman/config/settings.rb:33:in each’
/usr/share/foreman/config/settings.rb:33:in <top (required)>' /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in require’
/opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in require' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in block in require’
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:291:in load_dependency' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/dependencies.rb:324:in require’
/usr/share/foreman/config/application.rb:26:in <top (required)>' /opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in require’
/opt/rh/rh-ruby27/root/usr/share/rubygems/rubygems/core_ext/kernel_require.rb:83:in require' /usr/share/foreman/Rakefile:1:in <top (required)>’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/rake_module.rb:29:in load' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/rake_module.rb:29:in load_rakefile’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:703:in raw_load_rakefile' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:104:in block in load_rakefile’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in standard_exception_handling' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:103:in load_rakefile’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:82:in block in run' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:186:in standard_exception_handling’
/opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/lib/rake/application.rb:80:in run' /opt/rh/rh-ruby27/root/usr/share/gems/gems/rake-13.0.1/exe/rake:27:in <top (required)>’
/opt/rh/rh-ruby27/root/usr/bin/rake:23:in load' /opt/rh/rh-ruby27/root/usr/bin/rake:23:in ’
I believe the issue is in the file “/usr/share/foreman/config/settings.rb” in the below seciton, but i am unsure on exactly how to resolve. i suspect this is some issue related to a secondary organization
Force setting to true until all code using it is removed
[:locations_enabled, :organizations_enabled].each do |setting|
SETTINGS[setting] = true
end
Anyone can help me on this?
The issue was within the file “/etc/foreman/plugins/foreman-tasks.yaml” someone made a change to the file and did bad syntax which caused everything the process not start due to that config syntax issue.
