REX reports 'success' for failed remote commands

Problem:

Remote Execution feature (REX) is functioning, but it reports a ‘success’ exit status for remote commands even when they fail.

For example:

[root@foreman ~]# hammer job-invocation create --job-template "Run Command - Script Default" --inputs command="blabla" --search-query "name ~ myhost.mydomain.com"
Job invocation 52 created
[.................................................................................................................................................................] [100%]
1 task(s), 1 success, 0 fail
[root@foreman ~]# hammer job-invocation output --host myhost.mydomain.com --id 52
/var/tmp/foreman-ssh-cmd-d602e532-4db6-423c-9a16-34f183ffb4dd/script: line 1: blabla: command not found
Exit status: 0

The command blabla does not exist, so why is the job execution reported as successful?
The same behavior occurs when the command is run in the Web UI.

Expected outcome:

The exit status of a remote command should report the correct value:

  • Success (0): when no errors occured while executing the remote command
  • Failure (1): errors occured while executing the remote command

Foreman and Proxy versions:

[root@foreman ~]# hammer status
Version:           3.12.0
API Version:       v2
Database:
    Status:          ok
    Server Response: Duration: 0ms
Plugins:
 1) Name:    foreman-tasks
    Version: 9.2.3
 2) Name:    foreman_remote_execution
    Version: 13.2.5
 3) Name:    foreman_templates
    Version: 9.5.1
 4) Name:    katello
    Version: 4.14.0
Smart Proxies:
 1) Name:     foreman.<redacted>
    Version:  3.12.0
    Status:   ok
    Features:
     1) Name:    dynflow
        Version: 0.9.3
     2) Name:    pulpcore
        Version: 3.3.0
     3) Name:    script
        Version: 0.11.3
     4) Name:    templates
        Version: 3.12.0
     5) Name:    tftp
        Version: 3.12.0
     6) Name:    logs
        Version: 3.12.0
     7) Name:    registration
        Version: 3.12.0
Compute Resources:

candlepin:
    Status:          ok
    Server Response: Duration: 14ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 12ms
candlepin_events:
    Status:          ok
    message:         1 Processed, 0 Failed
    Server Response: Duration: 0ms
katello_events:
    Status:          ok
    message:         1 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp3:
    Status:          ok
    Server Response: Duration: 47ms
pulp3_content:
    Status:          ok
    Server Response: Duration: 47ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 3ms

Distribution and version:

CentOS Stream 9

What version of rubygem-smart_proxy_remote_execution_ssh do you have?

I have:

[root@foreman ~]# yum info rubygem-smart_proxy_remote_execution_ssh
Last metadata expiration check: 1:53:58 ago on Fri 01 Nov 2024 11:18:13 AM CET.
Installed Packages
Name         : rubygem-smart_proxy_remote_execution_ssh
Version      : 0.11.3
Release      : 1.fm3_12.el9
Architecture : noarch
Size         : 114 k
Source       : rubygem-smart_proxy_remote_execution_ssh-0.11.3-1.fm3_12.el9.src.rpm
Repository   : @System
From repo    : foreman-plugins
Summary      : Ssh remote execution provider for Foreman Smart-Proxy
URL          : https://github.com/theforeman/smart_proxy_remote_execution_ssh
License      : GPLv3
Description  : Ssh remote execution provider for Foreman Smart-Proxy.

This was fixed in 0.11.4, but this version didn’t land in 3.12 repos yet. I just opened packaging PRs[1,2] to make it happen.

[1] - Update rubygem-smart_proxy_remote_execution_ssh to 0.11.4 by adamruzicka · Pull Request #11433 · theforeman/foreman-packaging · GitHub
[2] - Update ruby-smart-proxy-remote-execution-ssh to 0.11.4 by adamruzicka · Pull Request #11434 · theforeman/foreman-packaging · GitHub

Great! Thanks a lot, Adam.