When I run a remote execution job on Foreman 3.10, the job will run successfully but after a timeout it gives this:
RuntimeError: Unexpected event #<Actions::ProxyAction::ProxyActionStopped: Actions::ProxyAction::ProxyActionStopped>
Expected outcome:
The command seems to work but gives that warning.
/var/log/foreman-proxy/proxy.log contains:
2024-08-14T15:49:34 10eb9531 [I] Finished GET /dynflow/tasks/19367fc3-b2a3-4eaf-a681-300ac448fff3/status with 200 (9.23 ms)
2024-08-14T15:49:34 132b82cf [E] Failed performing callback to Foreman server: 403 {
“error”: {“message”:“Access denied”,“details”:"Missing one of the required permissions: ",“missing_permissions”:}
}
That feels like some misconfiguration. What do the foreman-side logs say? You should be able to find a request to /foreman_tasks/api/tasks/callback which ended with 403 in there, could you please post it here?
Sorry, I misunderstood.
Ok, here’s the log. I’m guessing from it, I have to turn smart proxy off? I have to figure out how to do that if that’s the case
2024-08-16T12:36:33 [I|app|656a8d33] Completed 200 OK in 110ms (Views: 1.2ms | ActiveRecord: 6.0ms | Allocations: 12434)
2024-08-16T12:36:33 [I|app|bfa8c289] Started POST “/foreman_tasks/api/tasks/callback” for 10.4.13.157 at 2024-08-16 12:36:33 +0000
2024-08-16T12:36:33 [I|app|bfa8c289] Processing by ForemanTasks::Api::TasksController#callback as HTML
2024-08-16T12:36:33 [I|app|bfa8c289] Parameters: {“callback”=>{“task_id”=>“979d2be0-b783-49e2-97d4-736fde4d29f6”, “step_id”=>3}, “data”=>{“result”=>[{“output_type”=>“stdout”, “output”=>" % Total % Received % Xferd Average Speed Time Time Time Current\n Dload Upload Total Spent Left Speed\n\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0", “timestamp”=>1723811616.9880784}, {“output_type”=>“stdout”, “output”=>“\r 0 0 0 0 0 0 0 0 --:–:-- --:–:-- --:–:-- 0\r100 5447 100 5447 0 0 8743 0 --:–:-- --:–:-- --:–:-- 8729\nChecking dependencies\nChecking OS version… not recognized\nCreating adhd user\nuseradd: user ‘adhd’ already exists\nResolving user group "adhd" to group id: 1002\nDownloading and Installing adhd …\n”, “timestamp”=>1723811617.9378872}, {“output_type”=>“stdout”, “output”=>“Configuring adhd\n”, “timestamp”=>1723811666.5759661}, {“output_type”=>“stdout”, “output”=>“Setting adhd to start on boot\n”, “timestamp”=>1723811719.7032635}, {“output_type”=>“stdout”, “output”=>“Enabling adhd to be managed by systemd…\nboot-start enable command needs root privileges…\n”, “timestamp”=>1723811720.706551}, {“output_type”=>“stdout”, “output”=>“Found systemd config file /etc/systemd/system/adhd-edge.service, Do you want adhd to continue further and replace it with a new one (y/N)?:Exiting.\nmkdir: cannot create directory ‘/etc/systemd/system/adhd-edge.service.d’: File exists\nWarning: The unit file, source configuration file or drop-ins of adhd-edge.service changed on disk. Run ‘systemctl daemon-reload’ to reload units.\n”, “timestamp”=>1723811732.9341676}, {“output_type”=>“stdout”, “output”=>“Job for adhd-edge.service failed because a timeout was exceeded.\nSee "systemctl status adhd-edge.service" and "journalctl -xe" for details.\n”, “timestamp”=>1723811793.4164448}], “runner_id”=>“4ac0c58c-f77e-4b96-9a44-49c1a156a130”, “exit_status”=>0}, “task”=>{}}
2024-08-16T12:36:33 [W|app|bfa8c289] No smart proxy server found on [“redacted.company.com”, “redacted.company.com”] and is not in trusted_hosts
2024-08-16T12:36:33 [I|app|bfa8c289] Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 0.5ms | Allocations: 143)
2024-08-16T12:36:33 [I|app|bfa8c289] Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 0.9ms | Allocations: 235)
2024-08-16T12:36:33 [I|app|bfa8c289] Filter chain halted as #<Proc:0x000055defbd732e8 /usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14> rendered or redirected
2024-08-16T12:36:33 [I|app|bfa8c289] Completed 403 Forbidden in 15ms (Views: 2.4ms | ActiveRecord: 5.0ms | Allocations: 2220)
2024-08-16T12:36:34 [I|app|65052cbe] Started GET “/template_invocations/1257?since=1723811732.9341671&line_counter=17&=1723811616903" for 10.2.17.252 at 2024-08-16 12:36:34 +0000
2024-08-16T12:36:34 [I|app|65052cbe] Processing by TemplateInvocationsController#show as JS
2024-08-16T12:36:34 [I|app|65052cbe] Parameters: {“since”=>“1723811732.9341671”, “line_counter”=>“17”, "”=>“1723811616903”, “id”=>“1257”}
2024-08-16T12:36:34 [W|app|65052cbe] Scoped order is ignored, it’s forced to be batch order.
2024-08-16T12:36:34 [I|app|65052cbe] Rendered /usr/share/gems/gems/foreman_remote_execution-13.0.0/app/views/template_invocations/show.js.erb (Duration: 0.4ms | Allocations: 221)
2024-08-16T12:36:34 [I|app|65052cbe] Completed 200 OK in 98ms (Views: 1.2ms | ActiveRecord: 5.3ms | Allocations: 11315)
This is the thing you need to resolve. It seems that the ip address from which the callback comes (which is the smart proxy machine), resolves back to a different name than foreman expects, so foreman rejects the incoming request. One option would be to solve the naming situation by checking that everything resolves correctly, another would be to use a different name in the url of the smart proxy or add the other name (redacted.company.com in this case) to trusted hosts in settings as suggested by the logs.
