Job invocation problem with version 1.24.3

Support
1

Problem:
After upgrading Foreman packages from version 1.24.2 to 1.24.3, clicking any job from Monitor -> Jobs screen gives the error “undefined method `state’ for nil:NilClass”. Downgrading the version back to 1.24.2 would fix the issue.

Expected outcome:
Job details should be shown.

Foreman and Proxy versions:
1.24.3

Distribution and version:
Centos 7

Other relevant data:

production.log shows the following in debug mode:
2020-05-31T03:36:33 [W|for|db2fb8d3] Could not load execution plan 7df90211-51eb-4d9e-b50c-ae1b7068dde6 for task 84d9ce43-e9c1-4745-823
b-a9e8d91a1076
2020-05-31T03:36:33 [D|for|db2fb8d3] Backtrace for 'Could not load execution plan 7df90211-51eb-4d9e-b50c-ae1b7068dde6 for task 84d9ce4
3-e9c1-4745-823b-a9e8d91a1076' error (RuntimeError): The Dynflow world was not initialized yet. If your plugin uses it, make sure to ca
ll Rails.application.dynflow.require! in some initializer
 | /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/rails.rb:75:in `world'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/models/foreman_tasks/task/dynflow_task.rb:54:in `execution_pla
n'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/models/foreman_tasks/task/dynflow_task.rb:164:in `active_job?'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/models/foreman_tasks/task/dynflow_task.rb:145:in `main_action'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/models/foreman_tasks/task/dynflow_task.rb:196:in `get_humanize
d'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/models/foreman_tasks/task/dynflow_task.rb:133:in `humanized'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/views/foreman_tasks/tasks/dashboard/_latest_tasks_in_error_war
ning.html.erb:11:in `block in _d3f8fbcd40f2e1882a11bcfeb2b02d57'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/relation/delegation.rb:41:in `each'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/views/foreman_tasks/tasks/dashboard/_latest_tasks_in_error_war
ning.html.erb:9:in `_d3f8fbcd40f2e1882a11bcfeb2b02d57'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:159:in `block in render'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:170:in `instrument'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:354:in `instrument_render_template'
 | /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/template.rb:157:in `render'
 | /opt/theforeman/tfm/root/usr/share/gems/gems/deface-1.5.3/lib/deface/action_view_extensions.rb:43:in `render'
. . .
1 Like
2

Release notes for 1.24.3 mentions a change for:
Dynflow world gets shared across puma workers in production (#29258)

It seems that the changes by https://github.com/theforeman/foreman/pull/7487 were partially implemented in the package: https://yum.theforeman.org/releases/1.24/el7/x86_64/foreman-1.24.3-1.el7.noarch.rpm, and broke the installed feature. Change has been made in config/application.rb, but config/puma/production.rb is not included in the rpm package.

3

Thanks for digging in and debugging this issue!
@aruzicka can you take a look and see if we can somehow work around this or if we need to release 1.24.4 to fix it?

4

I spun up a fresh 1.24.3 machine and I’m failing to reproduce the issue there. As far as I can tell, on 1.24.3 we still run the rails process under passenger and dynflow gets initialized here https://github.com/theforeman/foreman/blob/develop/config/application.rb#L351 when the worker gets forked.

Since we’re not using puma there, having or not having config/puma/production.rb should have no effect.

Could we get full logs?

5

Thank you for looking into the issue. I am running puma. Attached log was extracted for the period when Foreman 1.24.3 was running.

6

I upgraded to version 1.24.3 again and can now work around by adding the file /usr/share/foreman/config/puma/production.rb:
mkdir /usr/share/foreman/config/puma
wget -O /usr/share/foreman/config/puma/production.rb https://github.com/theforeman/foreman/raw/19715ec794e5dca5aed738f9b9779c0daaf97250/config/puma/production.rb

The current 1.24-stable branch in github includes the updated config/application.rb but misses the new config/puma/production.rb made by the same https://github.com/theforeman/foreman/pull/7487 . I guess this is the root cause of the issue. 2.0-stable branch should suffer from the same issue but 2.1-stable branch should work well.

7

Thanks for digging into this @heanseng! I’ll make sure the correct cherry-picks are done for the stable branches, but since puma support in 1.24 is only experimental i don’t expect we’ll be releasing a new 1.24.4 with this change. 2.0.1 will already include it.

8

Thank you for the information, @tbrisker . I am happy with the workaround for now.

9

I am having a similar issue. I used foreman-installer --scenario katello with proxy on the same host.
Followed instruction to update the production.rb file.

My environment

CentOS Linux release 7.7.1908

katello.noarch                                                       3.14.1-1.el7
foreman.noarch                                                   1.24.3-1.el7
rubygem-smart_proxy_dynflow.noarch                0.2.4-1.el7
tfm-rubygem-dynflow.noarch                                1.4.2-1.fm1_24.el7
tfm-rubygem-smart_proxy_dynflow_core.noarch 0.2.2-1.fm1_22.el7

When I try to execute a simple task on a host an error is displayed as the following:

RuntimeError
The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.2/lib/dynflow/rails.rb:75:in `world'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/lib/foreman_tasks.rb:23:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/lib/foreman_tasks.rb:29:in `block in trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/lib/foreman_tasks.rb:49:in `block in rails_safe_trigger_task'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/lib/foreman_tasks.rb:48:in `rails_safe_trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/lib/foreman_tasks.rb:27:in `trigger_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/lib/foreman_tasks.rb:54:in `async_task'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.17.5/app/models/foreman_tasks/triggering.rb:65:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/models/job_invocation_composer.rb:366:in `trigger'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-2.0.8/app/controllers/job_invocations_controller.rb:43:in `create'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:194:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rendering.rb:30:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:136:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/callbacks.rb:41:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/rescue.rb:22:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `block in instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications/instrumenter.rb:23:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/notifications.rb:168:in `instrument'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/instrumentation.rb:32:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal/params_wrapper.rb:256:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activerecord-5.2.1/lib/active_record/railties/controller_runtime.rb:24:in `process_action'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/abstract_controller/base.rb:134:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionview-5.2.1/lib/action_view/rendering.rb:32:in `process'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:191:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_controller/metal.rb:252:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:52:in `dispatch'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:34:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:52:in `block in serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/journey/router.rb:35:in `serve'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/routing/route_set.rb:840:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/lib/katello/middleware/event_daemon.rb:10:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/static_dispatcher.rb:65:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/extractor/recorder.rb:137:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.14/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
/usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/tempfile_reaper.rb:15:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/etag.rb:25:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/conditional_get.rb:38:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/http/content_security_policy.rb:18:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:232:in `context'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/session/abstract/id.rb:226:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/cookies.rb:670:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:28:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/callbacks.rb:98:in `run_callbacks'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/debug_exceptions.rb:61:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:38:in `call_app'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/rack/logger.rb:28:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call'
/usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/request_id.rb:27:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.14.1/lib/katello/prevent_json_parsing.rb:12:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/method_override.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/runtime.rb:22:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/activesupport-5.2.1/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/executor.rb:14:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/actionpack-5.2.1/lib/action_dispatch/middleware/static.rb:127:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/sendfile.rb:111:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/engine.rb:524:in `call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `public_send'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/railties-5.2.1/lib/rails/railtie.rb:190:in `method_missing'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:68:in `block in call'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `each'
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/urlmap.rb:53:in `call'
/usr/share/ruby/vendor_ruby/phusion_passenger/rack/thread_handler_extension.rb:97:in `process_request'
/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:157:in `accept_and_process_next_request'
/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler/thread_handler.rb:110:in `main_loop'
/usr/share/ruby/vendor_ruby/phusion_passenger/request_handler.rb:415:in `block (3 levels) in start_threads'
/usr/share/ruby/vendor_ruby/phusion_passenger/utils.rb:113:in `block in create_thread_and_abort_on_exception'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Am I missing any other configurations?

10

Judging from the logs you are using passenger instead of puma, so the instructions in Job invocation problem with version 1.24.3 should have no effect.

I’m failing the issue reproduce on a fresh machine. Is there by any chance anything else you did?

11

Thanks for checking, Besides running the installer we share postgres DB with other application on the same host. I do see Not sure if the SELinux matters but we run in enforcing mode.

Here is the complete command used to install it. I will try on a vanilla CentOS 7 machine and see if the issue exist.

foreman-installer \
--scenario katello \
--log-level=DEBUG \
--verbose \
--verbose-log-level=DEBUG \
--foreman-proxy-log-level=DEBUG \
--foreman-db-password='password' \
--foreman-proxy-bind-host=0.0.0.0 \
--foreman-proxy-dhcp=true \
--foreman-proxy-dhcp-search-domains=fiftyone.example.com \
--foreman-proxy-dhcp-range='172.16.51.200 172.16.51.239' \
--foreman-proxy-dhcp-manage-acls=false \
--foreman-proxy-dhcp-gateway=172.16.51.1 \
--foreman-proxy-dns=true \
--foreman-proxy-puppetrun-provider=customrun \
--foreman-proxy-customrun-args=-y \
--foreman-proxy-customrun-cmd=/usr/local/sbin/puppetrun.sh \
--foreman-proxy-bmc=true \
--foreman-initial-admin-username=admin \
--foreman-initial-admin-password='changeme' \
--enable-foreman-cli \
--enable-foreman-cli-ansible \
--enable-foreman-compute-libvirt \
--enable-foreman-plugin-default-hostgroup \
--enable-foreman-plugin-hooks \
--enable-foreman-plugin-remote-execution \
--enable-foreman-plugin-ansible \
--enable-foreman-proxy-plugin-ansible \
--foreman-proxy-plugin-ansible-manage-runner-repo=false \
--enable-foreman-cli-openscap \
--enable-foreman-plugin-openscap \
--enable-foreman-proxy-plugin-openscap
12

I took your suggestion and ran the above mentioned command on a vanilla CentOS 7.7. After the install completed the foreman_tasks status is in failed state.

#hammer ping
database:       
    Status:          ok
    Server Response: Duration: 0ms
candlepin:      
    Status:          ok
    Server Response: Duration: 16ms
candlepin_auth: 
    Status:          ok
    Server Response: Duration: 12ms
pulp:           
    Status:          ok
    Server Response: Duration: 48ms
pulp_auth:      
    Status:          ok
    Server Response: Duration: 35ms
foreman_tasks:  
    Status:          FAIL
    Server Response: Message: The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer
13

We added some debug logging to config/application.rb. Turns out on this line[1], the value passed to the block is false and so dynflow doesn’t get initialized. The condition is there to only initialize dynflow once when passenger forks off the worker and on all instances I’ve seen so far, this worked the way it should. Anyone has any ideas?

[1] - https://github.com/theforeman/foreman/blob/develop/config/application.rb#L353

14

Thank you all for helping me troubleshoot the issue. Special thanks to @aruzicka for doing an install and providing me steps to debug the issue.
Turns out, our install differ on few packages and the one that stands out are mod_passenger and postgres. I wasn’t aware that our gold images contained different repos for the mentioned two packages. Once I reconfigured the yum repos and ran the installer the foreman_tasks started working as expected.

Again, thank you for looking into it and your assistance helped us figure out the issue.

1 Like
15

Glad to hear you got it working.

I know that when we’re testing RCs one of the sanity checks is that mod_passenger and tfm-rubygem-passenger are from the same repo ( foreman ) and in the same version. I never new a mismatch in this could cause such strange behavior. Kudos to you for figuring it out

1 Like