Problem:
When I run a remote job on a host (through remote execution plugin), the job completes successfully on the host but hangs and then shows up as failed in the WebUI. I have to mention that if I use, for example, yum to install a package, the web console will display the expected output.
Expected outcome:
The job should show up as successful in the WebUI.
Foreman and Proxy versions:
I am running Foreman 1.22 + Katello 3.12
Foreman and Proxy plugin versions:
Katello 3.12 and Foreman Remote Execution plugin 1.8.2
Other relevant data:
The job fails in the WebUI with the following message:
1:Failed to initialize: Foreman::Exception - ERF42-9239 [Foreman::Exception]: ERF42-5163 [Foreman::Exception]: The smart proxy task db7aaa93-d7b2-4d4c-912b-81e3f71b2935 failed.
could you alaborate a bit more on the job you are running? Maybe even share the Job templated (rendered or raw)?
How many hosts did you run the job on?
We have had similar issues in the past when running jobs that produced a lot of output on a large number of hosts (say several hundreds hosts with each aroud 1000 lines of output). I thought this would have been solved by now, but maybe that could still be the case .
I tried running just simple jobs (using “Run command - SSH default” job template) like echo to a file or “yum install -y” in my test environment. I tried running the job from the Host View > Schedule remote job (one host at a time).
I’ve read that thread, but unfortunantely, I wasn’t able to solve my issue. However I have managed to extract the logs from /var/log/foreman-proxy/smart_proxy_dynflow_core.log: https://pastebin.com/jFHN9SbS
The only interesting thing from that is: Run SmartProxyDynflowCore::Callback::Action403 Forbidden (RestClient::Forbidden)
I don’t really know what that means though.
When the job finishes, the smart proxy tries to call back to foreman. It seems this request is reaching Foreman, but is forbidden. Look for any 403s in /var/log/foreman/production.log. There should be a hint why it was forbidden.
I’ve managed to get it working by disabling “Require SSL for smart proxies” setting in Administer > Settings >Authentication . It seems that the ssl certificates weren’t properly configured in /etc/smart_proxy_dynflow_core/settings.yaml . Thank you for your support!
