Problem: I’ve been Foreman and Remote Execution this to patch and reboot many machines in a staggered time-span. Since I just upgraded to Foreman 1.23.0, jobs run all hosts almost immediately.
When viewing the Job invocation status as the job is running, it shows most of the hosts as Pending, but I can actually click on the host name and see the console output. The timestamp on the console output proves the job ran on all hosts almost immediately.
The job status correctly shows the schedule to be “Set to distribute over: 900 seconds”.
Expected outcome: Job would execute on hosts in a staggered fashion, timed evenly over the specified time-span.
Foreman and Proxy versions: Foreman 1.23.0
Foreman and Proxy plugin versions: Remote Execution 1.8.2
Other relevant data:
Example job invocation:
hammer job-invocation create --time-span 600 --job-template "Run Command - SSH Default" --inputs command="date; ls -la" --search-query "name ^ ( [25 hosts truncated] )" --async
I started the job and all 25 hosts completed faster than I could load the page, yet all host but the first was status: pending. The output of the date command shows they all executed about 20 seconds after I initiated the job or 1 min 20 seconds.
The job task clearly shows the time-span, and the task status has everything pending, but the job actually ran. This is all running from the locally installed proxy on the same server.
{
"job_invocation": {
"id": 436,
"name": "Commands",
"description": "Run date; ls -la"
},
"concurrency_control": {
"time": {
"tickets": null,
"free": 1,
"meta": {
"interval": 24,
"time_span": 600
}
}
},
"job_category": "Commands",
"job_invocation_id": 436,
"current_request_id": null,
"current_timezone": "Central Time (US & Canada)",
"current_user_id": 4,
"current_organization_id": null,
"current_location_id": null
}
Raw output:
{
"planned_count": 25,
"cancelled_count": 0,
"total_count": 25,
"failed_count": 0,
"pending_count": 24,
"success_count": 1
}
I’ve searched for logs, enabled debug logs and I’m not finding posts with a similar issue. Any idea how I can try to troubleshoot this?
Thank you!
Aaron
