Problem:
Remote Execution job to run Ansible Roles failing after upgrading from Foreman/Katello 2.1/3.16 to 2.2/3.17 when using sudo escalation WITH password. Job ends with:
fatal: [target.host]: FAILED! => { “msg”: “Missing sudo password” }
This worked prior to the upgrade by setting remote_execution_sudo_password. It appears that the db:migrate for remote_execution_sudo_password -> remote_execution_effective_user_password ran successfully, but I went ahead and re-set it anyway to no avail. It just appears that the runner isn’t supplying the password for sudo at all.
Expected outcome:
Job executes successfully by providing SSH and sudo password when running
Foreman and Proxy versions:
foreman and foreman-proxy 2.2.1 running on CentOS 7
Foreman and Proxy plugin versions:
- ansible-runner-1.4.6-1.el7.noarch
- tfm-rubygem-foreman_ansible-6.0.0-1.fm2_2.el7.noarch
- tfm-rubygem-foreman_ansible_core-3.0.4-1.fm2_2.el7.noarch
- tfm-rubygem-foreman_remote_execution-4.1.0-1.fm2_2.el7.noarch
- tfm-rubygem-foreman_remote_execution_core-1.3.1-1.el7.noarch
Distribution and version:
CentOS Linux release 7.9.2009 (Core)
Other relevant data:
ansible-playbook 2.9.15
config file = /usr/share/foreman-proxy/.ansible.cfg
configured module search path = [u’/usr/share/foreman-proxy/.ansible/plugins/modules’, u’/usr/share/ansible/plugins/modules’]
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible-playbook
python version = 2.7.5 (default, Nov 16 2020, 22:23:17) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
Using /usr/share/foreman-proxy/.ansible.cfg as config file
host_list declined parsing /tmp/d20201209-12177-1waaldu/inventory/hosts as it did not pass its verify_file() method
Parsed /tmp/d20201209-12177-1waaldu/inventory/hosts inventory source with script plugin
PLAYBOOK: playbook.yml *********************************************************
1 plays in playbook.yml
PLAY [all] *********************************************************************
TASK [Gathering Facts] *********************************************************
task path: /tmp/d20201209-12177-1waaldu/project/playbook.yml:2
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'echo ~svc-remoteexec && sleep 0'"'"''
<remote.server> (0, '/home/svc-remoteexec\n', '')
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/svc-remoteexec/.ansible/tmp `"&& mkdir "` echo /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433 `" && echo ansible-tmp-1607538373.43-7696-22467728853433="` echo /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433 `" ) && sleep 0'"'"''
<remote.server> (0, 'ansible-tmp-1607538373.43-7696-22467728853433=/home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433\n', '')
<remote.server> Attempting python interpreter discovery
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<remote.server> (0, 'PLATFORM\nLinux\nFOUND\n/usr/bin/python\n/usr/bin/python2.7\n/usr/libexec/platform-python\n/usr/bin/python\nENDFOUND\n', '')
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<remote.server> (0, '{"osrelease_content": "NAME=\\"CentOS Linux\\"\\nVERSION=\\"7 (Core)\\"\\nID=\\"centos\\"\\nID_LIKE=\\"rhel fedora\\"\\nVERSION_ID=\\"7\\"\\nPRETTY_NAME=\\"CentOS Linux 7 (Core)\\"\\nANSI_COLOR=\\"0;31\\"\\nCPE_NAME=\\"cpe:/o:centos:centos:7\\"\\nHOME_URL=\\"https://www.centos.org/\\"\\nBUG_REPORT_URL=\\"https://bugs.centos.org/\\"\\n\\nCENTOS_MANTISBT_PROJECT=\\"CentOS-7\\"\\nCENTOS_MANTISBT_PROJECT_VERSION=\\"7\\"\\nREDHAT_SUPPORT_PRODUCT=\\"centos\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\"7\\"\\n\\n", "platform_dist_result": ["centos", "7.9.2009", "Core"]}\n', '')
Using module file /usr/lib/python2.7/site-packages/ansible/modules/system/setup.py
<remote.server> PUT /tmp/ansible-local-7685B2SZDN/tmpdlWegS TO /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/AnsiballZ_setup.py
<remote.server> SSH: EXEC sshpass -d8 sftp -o BatchMode=no -b - -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 '[remote.server]'
<remote.server> (0, 'sftp> put /tmp/ansible-local-7685B2SZDN/tmpdlWegS /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/AnsiballZ_setup.py\n', '')
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'chmod u+x /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/ /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/AnsiballZ_setup.py && sleep 0'"'"''
<remote.server> (0, '', '')
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 -tt remote.server '/bin/sh -c '"'"'sudo -H -S -n -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-jigrjdoviuzdpisumtyuidilgfzyuyiw ; /usr/bin/python /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/AnsiballZ_setup.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation requires password
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'rm -f -r /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/ > /dev/null 2>&1 && sleep 0'"'"''
<remote.server> (0, '', '')
fatal: [remote.server]: FAILED! => {
"msg": "Missing sudo password"
}
PLAY RECAP *********************************************************************
remote.server : ok=0 changed=0 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0