"Proxy task gone missing from the smart proxy" if job takes more than 10m to finish

Problem:
Running a standard job of type script just executing the sleep command. After 10m the job fails with message “Actions::ProxyAction::ProxyActionMissing: Proxy task gone missing from the smart proxy”. Although foreman marks the job has failed the script continues to run and I’m able to follow the execution in /var/tmp/foreman-ssh-cmd-<id>/output file.

The 10m (600 secs) come from the “POLL_INTERVAL = 600” constant in watch_delegated_proxy_sub_tasks.rb. I tried to do some debugging and discovered that the condition is triggered in the function process_task_results because the state of the parent job is returning as stopped (“state”=>“stopped”, “result”=>“success”), hence the job is considered missing. I don’t find the corresponding parent_task_id in the foreman_tasks_tasks table.
I’m trying to follow the code but I’m hoping someone here that is more knowledgeable about both ruby and the foreman code might help get to a conclusion a lot quicker.

Expected outcome:
Job to finish.

Foreman and Proxy versions:
foreman-proxy-3.9.1-1.el8

Foreman and Proxy plugin versions:
rubygem-foreman-tasks-9.0.1-1.fm3_9.el8

Distribution and version:
RHEL 8.9

Thanks!

Hi,
you’re not the first to run into this, see Failed tasks to update packages - proxy task gone missing - #20 by m-bucher for more details.

Well done on the analysis, you got pretty far in there

Thanks, that fix works for me as well!