Ansible Remote Execution with sudo password broken after 2.1/3.16 to 2.2/3.17 upgrade

Matthew_Hyclak · December 9, 2020, 6:50pm

Problem:
Remote Execution job to run Ansible Roles failing after upgrading from Foreman/Katello 2.1/3.16 to 2.2/3.17 when using sudo escalation WITH password. Job ends with:

fatal: [target.host]: FAILED! => { “msg”: “Missing sudo password” }

This worked prior to the upgrade by setting remote_execution_sudo_password. It appears that the db:migrate for remote_execution_sudo_password -> remote_execution_effective_user_password ran successfully, but I went ahead and re-set it anyway to no avail. It just appears that the runner isn’t supplying the password for sudo at all.

Expected outcome:

Job executes successfully by providing SSH and sudo password when running

Foreman and Proxy versions:

foreman and foreman-proxy 2.2.1 running on CentOS 7

Foreman and Proxy plugin versions:

ansible-runner-1.4.6-1.el7.noarch
tfm-rubygem-foreman_ansible-6.0.0-1.fm2_2.el7.noarch
tfm-rubygem-foreman_ansible_core-3.0.4-1.fm2_2.el7.noarch
tfm-rubygem-foreman_remote_execution-4.1.0-1.fm2_2.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.3.1-1.el7.noarch

Distribution and version:

CentOS Linux release 7.9.2009 (Core)

Other relevant data:

ansible-playbook 2.9.15
config file = /usr/share/foreman-proxy/.ansible.cfg
configured module search path = [u’/usr/share/foreman-proxy/.ansible/plugins/modules’, u’/usr/share/ansible/plugins/modules’]
ansible python module location = /usr/lib/python2.7/site-packages/ansible
executable location = /usr/bin/ansible-playbook
python version = 2.7.5 (default, Nov 16 2020, 22:23:17) [GCC 4.8.5 20150623 (Red Hat 4.8.5-44)]
Using /usr/share/foreman-proxy/.ansible.cfg as config file
host_list declined parsing /tmp/d20201209-12177-1waaldu/inventory/hosts as it did not pass its verify_file() method
Parsed /tmp/d20201209-12177-1waaldu/inventory/hosts inventory source with script plugin

PLAYBOOK: playbook.yml *********************************************************
1 plays in playbook.yml

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
task path: /tmp/d20201209-12177-1waaldu/project/playbook.yml:2
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'echo ~svc-remoteexec && sleep 0'"'"''
<remote.server> (0, '/home/svc-remoteexec\n', '')
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/svc-remoteexec/.ansible/tmp `"&& mkdir "` echo /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433 `" && echo ansible-tmp-1607538373.43-7696-22467728853433="` echo /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433 `" ) && sleep 0'"'"''
<remote.server> (0, 'ansible-tmp-1607538373.43-7696-22467728853433=/home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433\n', '')
<remote.server> Attempting python interpreter discovery
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'echo PLATFORM; uname; echo FOUND; command -v '"'"'"'"'"'"'"'"'/usr/bin/python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python3.5'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.7'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python2.6'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/libexec/platform-python'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'/usr/bin/python3'"'"'"'"'"'"'"'"'; command -v '"'"'"'"'"'"'"'"'python'"'"'"'"'"'"'"'"'; echo ENDFOUND && sleep 0'"'"''
<remote.server> (0, 'PLATFORM\nLinux\nFOUND\n/usr/bin/python\n/usr/bin/python2.7\n/usr/libexec/platform-python\n/usr/bin/python\nENDFOUND\n', '')
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'/usr/bin/python && sleep 0'"'"''
<remote.server> (0, '{"osrelease_content": "NAME=\\"CentOS Linux\\"\\nVERSION=\\"7 (Core)\\"\\nID=\\"centos\\"\\nID_LIKE=\\"rhel fedora\\"\\nVERSION_ID=\\"7\\"\\nPRETTY_NAME=\\"CentOS Linux 7 (Core)\\"\\nANSI_COLOR=\\"0;31\\"\\nCPE_NAME=\\"cpe:/o:centos:centos:7\\"\\nHOME_URL=\\"https://www.centos.org/\\"\\nBUG_REPORT_URL=\\"https://bugs.centos.org/\\"\\n\\nCENTOS_MANTISBT_PROJECT=\\"CentOS-7\\"\\nCENTOS_MANTISBT_PROJECT_VERSION=\\"7\\"\\nREDHAT_SUPPORT_PRODUCT=\\"centos\\"\\nREDHAT_SUPPORT_PRODUCT_VERSION=\\"7\\"\\n\\n", "platform_dist_result": ["centos", "7.9.2009", "Core"]}\n', '')
Using module file /usr/lib/python2.7/site-packages/ansible/modules/system/setup.py
<remote.server> PUT /tmp/ansible-local-7685B2SZDN/tmpdlWegS TO /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/AnsiballZ_setup.py
<remote.server> SSH: EXEC sshpass -d8 sftp -o BatchMode=no -b - -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 '[remote.server]'
<remote.server> (0, 'sftp> put /tmp/ansible-local-7685B2SZDN/tmpdlWegS /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/AnsiballZ_setup.py\n', '')
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'chmod u+x /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/ /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/AnsiballZ_setup.py && sleep 0'"'"''
<remote.server> (0, '', '')
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 -tt remote.server '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-jigrjdoviuzdpisumtyuidilgfzyuyiw ; /usr/bin/python /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/AnsiballZ_setup.py'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation requires password
<remote.server> ESTABLISH SSH CONNECTION FOR USER: svc-remoteexec
<remote.server> SSH: EXEC sshpass -d8 ssh -o ProxyCommand=none -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy"' -o 'User="svc-remoteexec"' -o ConnectTimeout=10 -o ControlPath=/var/lib/foreman-proxy/ansible/cp/35fac8a072 remote.server '/bin/sh -c '"'"'rm -f -r /home/svc-remoteexec/.ansible/tmp/ansible-tmp-1607538373.43-7696-22467728853433/ > /dev/null 2>&1 && sleep 0'"'"''
<remote.server> (0, '', '')
fatal: [remote.server]: FAILED! => {
    "msg": "Missing sudo password"
}
PLAY RECAP *********************************************************************
remote.server : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Marek_Hulan · December 14, 2020, 9:28am

This could have been caused by https://github.com/theforeman/foreman_ansible/commit/8570ca3a543826ed29541d2508a974cfffa2ca63

aruzicka · December 14, 2020, 9:48am

We missed the core part when doing a release, it should be fixed in foreman_ansible_core-4.0.0

Matthew_Hyclak · December 14, 2020, 2:55pm

Glad it’s something simple. Is there a plan for releasing foreman_ansible_core-4.0.0 for Foreman 2.2? I tried grabbing the plugin rpms for ansible, ansible_core, remote_execution and remote_execution_core, but they didn’t play nice with the rest of the system - so I presume they’re built against 2.3.

Thanks!

Matthew_Hyclak · January 13, 2021, 8:03pm

@aruzicka Will the updated foreman_ansible_core be released with 2.2.3? Or is this going to remain broken for all of 2.2.x? I didn’t see anything mentioned in the meeting notes for the 2.2.3 release.

Thanks,
Matt

aruzicka · January 15, 2021, 10:46am

Updated foreman_ansible_core will be eventually released for 2.2.z, but I first have to check if we can just take the entire release or if I’ll need to cherry-pick only selected bits.