Remote execution dynflow not initialized yet

gertz22 · July 26, 2023, 12:08pm

Problem:
I want try using remote execution but when I try submit job then I got this error:
The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer
Expected outcome:
succesfully execute Job
Foreman and Proxy versions:
Foreman 3.7.0
Foreman and Proxy plugin versions:
foreman-tasks 8.0.2
foreman_puppet 5.1.2
foreman_remote_execution 10.0.1

Distribution and version:
Debian 11.7

aruzicka · July 26, 2023, 2:04pm

How exactly did you install Foreman?

I just tried deploying a fresh instance and it seems to work out of the box.

gertz22 · July 26, 2023, 2:36pm

I run foreman-installer
and after that:
foreman-installer --enable-foreman-plugin-proxmox --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-script

nothing else

aruzicka · July 26, 2023, 2:59pm

That should be be fine. All this dynflow stuff should be set up when puma workers boot so that’s most likely the place to look at. What does systemctl status foreman and journalctl -u foreman give you?

gertz22 · July 26, 2023, 3:02pm

systemctl status foreman

     Loaded: loaded (/lib/systemd/system/foreman.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/foreman.service.d
             └─installer.conf
     Active: active (running) since Wed 2023-07-26 15:48:15 CEST; 1h 12min ago
TriggeredBy: ● foreman.socket
       Docs: https://theforeman.org
   Main PID: 3603 (foreman-ruby)
     Status: "Puma 6.3.0: worker: { 5/5 threads, 5 available, 0 backlog }"
      Tasks: 16 (limit: 2318)
     Memory: 289.9M
        CPU: 26.642s
     CGroup: /system.slice/foreman.service
             └─3603 puma 6.3.0 (unix:///run/foreman.sock) [foreman]

Jul 26 15:48:15 foreman.feel4.cz foreman[3603]: * Puma version: 6.3.0 (ruby 2.7.4-p191) ("Mugi No Toki Itaru")
Jul 26 15:48:15 foreman.feel4.cz foreman[3603]: *  Min threads: 5
Jul 26 15:48:15 foreman.feel4.cz foreman[3603]: *  Max threads: 5
Jul 26 15:48:15 foreman.feel4.cz foreman[3603]: *  Environment: production
Jul 26 15:48:15 foreman.feel4.cz foreman[3603]: *          PID: 3603
Jul 26 15:48:15 foreman.feel4.cz foreman[3603]: * Activated unix:///run/foreman.sock
Jul 26 15:48:15 foreman.feel4.cz foreman[3603]: * Starting control server on unix:///usr/share/foreman/tmp/sockets/pumactl.sock
Jul 26 15:48:15 foreman.feel4.cz foreman[3603]: Sending status to systemd every 1.0 sec
Jul 26 15:48:15 foreman.feel4.cz foreman[3603]: Use Ctrl-C to stop
Jul 26 15:48:15 foreman.feel4.cz systemd[1]: Started Foreman.

Jul 26 13:04:04 foreman.feel4.cz systemd[1]: foreman.service: Succeeded.
Jul 26 13:04:04 foreman.feel4.cz systemd[1]: Stopped Foreman.
Jul 26 13:04:04 foreman.feel4.cz systemd[1]: foreman.service: Consumed 13.064s CPU time.
Jul 26 13:04:04 foreman.feel4.cz systemd[1]: Starting Foreman...
Jul 26 13:04:08 foreman.feel4.cz foreman[34279]: => Booting Puma
Jul 26 13:04:08 foreman.feel4.cz foreman[34279]: => Rails 6.1.7.3 application starting in production
Jul 26 13:04:08 foreman.feel4.cz foreman[34279]: => Run `bin/rails server --help` for more startup options
Jul 26 13:04:16 foreman.feel4.cz foreman[34279]: * Enabling systemd notification integration
Jul 26 13:04:16 foreman.feel4.cz foreman[34279]: Puma starting in single mode...
Jul 26 13:04:16 foreman.feel4.cz foreman[34279]: * Puma version: 6.3.0 (ruby 2.7.4-p191) ("Mugi No Toki Itaru")
Jul 26 13:04:16 foreman.feel4.cz foreman[34279]: *  Min threads: 5
Jul 26 13:04:16 foreman.feel4.cz foreman[34279]: *  Max threads: 5
Jul 26 13:04:16 foreman.feel4.cz foreman[34279]: *  Environment: production
Jul 26 13:04:16 foreman.feel4.cz foreman[34279]: *          PID: 34279
Jul 26 13:04:16 foreman.feel4.cz foreman[34279]: * Activated unix:///run/foreman.sock
Jul 26 13:04:16 foreman.feel4.cz foreman[34279]: Sending status to systemd every 1.0 sec

aruzicka · July 26, 2023, 3:06pm

That somewhat differs from what I see on my box.

# systemctl status foreman
● foreman.service - Foreman
     Loaded: loaded (/lib/systemd/system/foreman.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/foreman.service.d
             └─installer.conf
     Active: active (running) since Wed 2023-07-26 14:55:49 UTC; 9min ago
TriggeredBy: ● foreman.socket
       Docs: https://theforeman.org
   Main PID: 26796 (foreman-ruby)
     Status: "Puma 6.3.0: cluster: 3/3, worker_status: [{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog },{ 5/5 threads, 5 available, 0 backlog }]"
      Tasks: 62 (limit: 14315)
     Memory: 704.5M
        CPU: 19.815s
     CGroup: /system.slice/foreman.service
             ├─26796 puma 6.3.0 (unix:///run/foreman.sock) [foreman]
             ├─27012 puma: cluster worker 0: 26796 [foreman]
             ├─27013 puma: cluster worker 1: 26796 [foreman]
             └─27014 puma: cluster worker 2: 26796 [foreman]

Jul 26 14:55:49 special-reptile foreman[26796]: [26796] *     Restarts: (✔) hot (✖) phased
Jul 26 14:55:49 special-reptile foreman[26796]: [26796] * Preloading application
Jul 26 14:55:49 special-reptile foreman[26796]: [26796] * Activated unix:///run/foreman.sock
Jul 26 14:55:49 special-reptile foreman[26796]: [26796] Use Ctrl-C to stop
Jul 26 14:55:49 special-reptile foreman[26796]: [26796] * Starting control server on unix:///usr/share/foreman/tmp/sockets/pumactl.sock
Jul 26 14:55:49 special-reptile foreman[26796]: [26796] Sending status to systemd every 1.0 sec
Jul 26 14:55:49 special-reptile foreman[26796]: [26796] - Worker 0 (PID: 27012) booted in 0.11s, phase: 0
Jul 26 14:55:49 special-reptile foreman[26796]: [26796] - Worker 2 (PID: 27014) booted in 0.11s, phase: 0
Jul 26 14:55:49 special-reptile foreman[26796]: [26796] - Worker 1 (PID: 27013) booted in 0.12s, phase: 0
Jul 26 14:55:49 special-reptile systemd[1]: Started Foreman.

What do you have in /etc/systemd/system/foreman.service.d/installer.conf?

gertz22 · July 26, 2023, 3:09pm

[Service]
User=foreman
Environment=FOREMAN_ENV=production
Environment=FOREMAN_HOME=/usr/share/foreman
Environment=FOREMAN_PUMA_THREADS_MIN=5
Environment=FOREMAN_PUMA_THREADS_MAX=5
Environment=FOREMAN_PUMA_WORKERS=-1

aruzicka · July 26, 2023, 3:21pm

The -1 for FOREMAN_PUMA_WORKERS is odd. Looking at where it comes from[1] I’d say your machine is a bit underpowered. You could probably try forcing it to something else by adding --foreman-foreman-service-puma-workers=1 to the installer, but ymmv. Getting a beefier machine would be better approach in the long run.

[1] - https://github.com/theforeman/puppet-foreman/blob/a4157ab77ed8589c3d312c13890a57b7b62d3599/manifests/config.pp#L78-L86C4

ekohl · July 26, 2023, 3:32pm

For manual input we do verify the data type with as Integer[0], which sets 0 as a lower bound. However, we don’t do so for the formula because we didn’t anticipate it.

evgeni · July 26, 2023, 3:34pm

But even then, 0 seems wrong, shouldn’t it be Integer[1]?

gertz22 · July 26, 2023, 3:48pm

So I have 6 CPU for virtual and 4GB of ram, after I run installer with puma workers=1 it started working. Im not sure if it right to let it run that way

evgeni · July 26, 2023, 3:55pm

with 4GB memory it should have created 2 workers. weird.
what’s the output of facter memory.system.total_bytes on your system?

gertz22 · July 26, 2023, 3:56pm

Output is: 4108808192

evgeni · July 26, 2023, 4:38pm

Hmm, that’s 3.8GiB, minus 1.5 makes 2.3, floored 2. Yepp, still should have done 2 workers. weird.

gertz22 · July 26, 2023, 5:27pm

if I will increase memory it will automaticaly increase workers after rerun foreman-insteller?

evgeni · July 27, 2023, 5:43am

Now that you passed --foreman-foreman-service-puma-workers it will not, no.
But you can reset the value with --reset-foreman-foreman-service-puma-workers, and then it will become dynamic again.

ekohl · July 27, 2023, 9:55am

0 workers means single (non-clustered) mode:

github.com

puma/puma/blob/master/docs/deployment.md#single-vs-cluster-mode

# Deployment engineering for Puma

Puma expects to be run in a deployed environment eventually. You can use it as
your development server, but most people use it in their production deployments.

To that end, this document serves as a foundation of wisdom regarding deploying
Puma to production while increasing happiness and decreasing downtime.

## Specifying Puma

Most people will specify Puma by including `gem "puma"` in a Gemfile, so we'll
assume this is how you're using Puma.

## Single vs. Cluster mode

Initially, Puma was conceived as a thread-only web server, but support for
processes was added in version 2.

To run `puma` in single mode (i.e., as a development environment), set the
number of workers to 0; anything higher will run in cluster mode.

This file has been truncated. show original

gertz22 · July 27, 2023, 10:19am

So thats mean in single mode you cant use foreman remote execute commnads and some more features? Until I force some workers?

tomm · August 6, 2023, 9:44am

after a while I figured i needed to not set FOREMAN_PUMA_WORKERS = >1 or foreman-installer --foreman-puma-workders with a value over 1, or not set it at all.
and boom, fresh deployment works with every tested version (did each from 3.2 to 3.7…)