Problem:
For a while now I have noticed that scheduling of remote execution jobs is very slow. For each host it sits there “Awaiting start” and goes through each host from there through planned - pending - running approx one host per second.
And then once 100 hosts are in “running” it does actually execute the commands (which I think it always did).
So it basically takes approx. 100 seconds until the first batch of 100 hosts runs and then it continues from there.
It was much faster before but I can’t really tell since when it is this way. I only really noticed lately when I had to run some scripts on hundreds of hosts.
Expected outcome:
Much faster scheduling and earlier start of execution.
Foreman and Proxy versions:
foreman-3.12.1-1.el9.noarch
katello-4.14.3-1.el9.noarch
rubygem-foreman_remote_execution-13.2.5-1.fm3_12.el9.noarch
rubygem-hammer_cli_foreman_remote_execution-0.3.0-1.el9.noarch
rubygem-smart_proxy_remote_execution_ssh-0.11.4-1.fm3_12.el9.noarch
That does indeed sound slow. On my nightly machine, I’m able to push out a command to 100 hosts end to end in 45 seconds as admin and in 53 seconds as a non-admin with all permissions. Not particularly great, but still much better than what you’re describing.
I wonder where it spends all that time. The process of planning for the hosts is strictly sequential and it does a lot in there. How are your sidekiq workers configured? Maybe dedicating a worker process to rex could help speeding things up?
It used to be different up until Foreman 1.22 iirc.
I also tested with 100 hosts, running “date”. Proxy batch size in settings is default 100.
The jobs started at 2025-03-13 11:48:47 +0100.
The first hosts printed Thu Mar 13 11:50:09 CET 2025.
The last hosts printed Thu Mar 13 11:50:18 CET 2025.
So it took 82s until the first hosts ran the script. All running with an admin account.
On the task page I can see that the remote action tasks “started at” March 13, 2025 at 11:48:48 AM from the first host through March 13, 2025 at 11:50:06 AM for the last one.
