Remote execution fails to use ssh key

George · May 4, 2022, 11:03am

Problem:
Remote execution doesn’t use ssh key and passphrase to login to target host and just gets stuck waiting.

/var/log/secure

May  4 13:10:30 foreman sshd[1015232]: Postponed publickey for foreman-remote from 123.123.123.123 port 34308 ssh2 [preauth]

So according to that remote execution is not even attempting to use the ssh key.

Config

cat /etc/foreman-proxy/settings.d/remote_execution_ssh.yml
---
:enabled: https
:ssh_identity_key_file: /var/lib/foreman-proxy/ssh/id_ecdsa
:local_working_dir: /var/tmp
:remote_working_dir: /var/tmp
:kerberos_auth: false

# Whether to run remote execution jobs asynchronously
:mode: ssh

#############################
Manual attempt with a key works. And also informs of failed logins so it has atleast attemped some ssh connection. But seeing the

ssh -i /var/lib/foreman-proxy/ssh/id_ecdsa foreman-remote@foreman.example.com
Enter passphrase for key '/var/lib/foreman-proxy/ssh/id_ecdsa':

UNAUTHORIZED ACCESS TO THIS DEVICE IS PROHIBITED

All activities performed on this device are logged and monitored.

Last failed login: Wed May  4 03:22:05 EEST 2022 from foreman.example.com on ssh:notty
There were 7 failed login attempts since the last successful login.
Last login: Tue May  3 15:41:47 2022 from foreman.example.com

/var/log/secure on manual attempt

May  4 11:03:57 foreman sshd[1008201]: Accepted publickey for foreman-remote from 123.123.123.123 port 57884 ssh2: 
May  4 11:03:57 foreman systemd[1008217]: pam_unix(systemd-user:session): session opened for user foreman-remote by (uid=0)
May  4 11:03:57 foreman sshd[1008201]: pam_unix(sshd:session): session opened for user foreman-remote by (uid=0)

Production log on the task.

2022-05-04T13:10:29 [I|app|d2d9732f]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"1/hdSyv7VN65n4r8ZNML+Tg9sEBomtn2FoYLc/IdmQ3JTecvdQov6FHJo4Fq1i7oDouKmm0duZmkJF31oG5WVw==", "job_invocation"=>{"job_category"=>"Packages", "remote_execution_feature_id"=>"", "providers"=>{"SSH"=>{"job_template_id"=>"143", "job_tem
plates"=>{"141"=>{"effective_user"=>"", "execution_timeout_interval"=>""}, "143"=>{"input_values"=>"[FILTERED]", "effective_user"=>"", "execution_timeout_interval"=>""}}}}, "description"=>"", "description_override"=>"%{action} package(s) %{package}", "description_format"=>"%{action} package(s) %{package}", "password
"=>"[FILTERED]", "key_passphrase"=>"redacted_since_this_was_plaintext", "effective_user_password"=>"[FILTERED]", "concurrency_level"=>"", "time_span"=>""}, "targeting"=>{"bookmark_id"=>"", "search_query"=>"name ^ (foreman.palvelu.local)", "randomized_ordering"=>"false", "targeting_type"=>"static_query"}, "fakepassword"=>"[FILTER
ED]", "triggering"=>{"mode"=>"immediate", "start_at_raw"=>"2022-05-04 13:10", "start_before_raw"=>"", "input_type"=>"daily", "cronline"=>"", "days"=>"", "days_of_week"=>{"1"=>"0", "2"=>"0", "3"=>"0", "4"=>"0", "5"=>"0", "6"=>"0", "7"=>"0"}, "time"=>{"time(1i)"=>"2022", "time(2i)"=>"5", "time(3i)"=>"4", "time(4i)"=>"
13", "time(5i)"=>"10"}, "max_iteration"=>"", "end_time_limited"=>"false", "end_time"=>{"end_time(1i)"=>"2022", "end_time(2i)"=>"5", "end_time(3i)"=>"4", "end_time(4i)"=>"13", "end_time(5i)"=>"10"}, "purpose"=>""}, "commit"=>"Submit"}

Expected outcome:
Remote execution uses ssh key and passphrase and
Foreman and Proxy versions:
foreman 3.2.0

foreman-tasks	The goal of this plugin is to unify the way of showing task statuses across the Foreman instance. It defines Task model for keeping the information about the tasks and Lock for assigning the tasks to resources. The locking allows dealing with preventing multiple colliding tasks to be run on the same resource. It also optionally provides Dynflow infrastructure for using it for managing the tasks.	Ivan Nečas	6.0.1
foreman_puppet	Allow assigning Puppet environments and classes to the Foreman Hosts.	Ondřej Ezr and Shira Maximov	3.0.5
foreman_remote_execution	A plugin bringing remote execution to the Foreman, completing the config management functionality with remote management functionality.	Foreman Remote Execution team	6.0.0
katello	Katello adds Content and Subscription Management to Foreman. For this it relies on Candlepin and Pulp.	N/A	4.4.0.2

candlepin-4.1.10-1.el8.noarch
candlepin-selinux-4.1.10-1.el8.noarch
foreman-3.2.0-1.el8.noarch
foreman-cli-3.2.0-1.el8.noarch
foreman-debug-3.2.0-1.el8.noarch
foreman-dynflow-sidekiq-3.2.0-1.el8.noarch
foreman-installer-3.2.0-1.el8.noarch
foreman-installer-katello-3.2.0-1.el8.noarch
foreman-postgresql-3.2.0-1.el8.noarch
foreman-proxy-3.2.0-1.el8.noarch
foreman-release-3.2.0-1.el8.noarch
foreman-selinux-3.2.0-1.el8.noarch
foreman-service-3.2.0-1.el8.noarch
foreman.palvelu.local-apache-1.0-2.noarch
foreman.palvelu.local-foreman-client-1.0-1.noarch
foreman.palvelu.local-foreman-proxy-1.0-2.noarch
foreman.palvelu.local-foreman-proxy-client-1.0-1.noarch
foreman.palvelu.local-puppet-client-1.0-1.noarch
katello-4.4.0-1.el8.noarch
katello-ca-consumer-foreman.palvelu.local-1.0-2.noarch
katello-certs-tools-2.8.2-1.el8.noarch
katello-client-bootstrap-1.7.8-1.el8.noarch
katello-common-4.4.0-1.el8.noarch
katello-debug-4.4.0-1.el8.noarch
katello-default-ca-1.0-1.noarch
katello-repos-4.4.0-1.el8.noarch
katello-selinux-4.0.2-1.el8.noarch
katello-server-ca-1.0-2.noarch
pulp-client-1.0-1.noarch
pulpcore-selinux-1.3.0-1.el8.x86_64
python38-pulp-ansible-0.10.1-1.el8.noarch
python38-pulp-certguard-1.5.1-1.el8.noarch
python38-pulp-container-2.9.2-1.el8.noarch
python38-pulp-deb-2.16.1-1.el8.noarch
python38-pulp-file-1.10.1-1.el8.noarch
python38-pulp-python-3.5.2-1.el8.noarch
python38-pulp-rpm-3.17.3-2.el8.noarch
python38-pulpcore-3.16.7-1.el8.noarch
qpid-proton-c-0.35.0-1.el8.x86_64
rubygem-foreman-tasks-6.0.1-1.fm3_2.el8.noarch
rubygem-foreman_maintain-1.0.3-1.el8.noarch
rubygem-foreman_puppet-3.0.5-1.fm3_2.el8.noarch
rubygem-foreman_remote_execution-6.0.0-1.fm3_2.el8.noarch
rubygem-hammer_cli-3.2.0-1.20220214173651git27087bf.el8.noarch
rubygem-hammer_cli_foreman-3.2.0-1.20220214175116git323f240.el8.noarch
rubygem-hammer_cli_foreman_puppet-0.0.4-1.fm3_1.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.2-1.fm3_0.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.17-1.fm3_2.el8.noarch
rubygem-hammer_cli_katello-1.4.1-1.el8.noarch
rubygem-katello-4.4.0.2-2.el8.noarch
rubygem-pulp_ansible_client-0.10.1-1.el8.noarch
rubygem-pulp_certguard_client-1.5.0-1.el8.noarch
rubygem-pulp_container_client-2.9.0-1.el8.noarch
rubygem-pulp_deb_client-2.16.0-1.el8.noarch
rubygem-pulp_file_client-1.10.0-1.el8.noarch
rubygem-pulp_ostree_client-2.0.0-0.1.a1.el8.noarch
rubygem-pulp_python_client-3.5.2-1.el8.noarch
rubygem-pulp_rpm_client-3.17.4-1.el8.noarch
rubygem-pulpcore_client-3.16.0-1.el8.noarch
rubygem-qpid_proton-0.35.0-1.el8.x86_64
rubygem-smart_proxy_pulp-3.2.0-2.fm3_2.el8.noarch

Other relevant data:
I set the passphrase from the ui and did not set ssh password from there as it was not goint to be used anyway.

aruzicka · May 4, 2022, 12:02pm

Hi,
am I reading it right that you have specified both key passphrase and a password? It is a little bit hard to tell since bits of the log are redacted

George · May 4, 2022, 12:36pm

I only specified a key passphrase. No password should be set for remote execution

George · May 4, 2022, 12:39pm

All though it’s possible that a password was set earlier in some early tests months ago.

aruzicka · May 4, 2022, 12:40pm

Could be, it seems to work on my machine if I provide only the passphrase. If I provide both (or none) and use a key with passphrase, then it breaks.

George · May 4, 2022, 12:43pm

What’s the easiest way to purge possible old settings regarding this? Some hammer command?

George · May 5, 2022, 8:18am

Left a job running for the night to see if what would happen. Turns out it failed during the night when it couldn’t login with a password. Which it shouldn’t do.

May  5 03:17:03 foreman sshd[1056072]: pam_access(sshd:auth): access denied for user `foreman-remote' from `foreman.example.com'
May  5 03:17:03 foreman sshd[1056072]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=foreman.example.com user=foreman-remote
May  5 03:17:03 foreman sshd[1056072]: pam_sss(sshd:auth): received for user foreman-remote: 9 (Authentication service cannot retrieve authentication info)
May  5 03:17:05 foreman sshd[1056072]: Failed password for foreman-remote from 123.123.123.123 port 53858 ssh2

Tried setting the password to default using this. And then setting the keyphrase and proceeding to restart foreman-proxy.service.

hammer settings set --name remote_execution_effective_user_password --value ""

Setting [remote_execution_effective_user_password] updated to [*****].


hammer settings set --name remote_execution_ssh_key_passphrase --value "redacted_info"

Setting [remote_execution_ssh_key_passphrase] updated to [*****].

Running install package cowsay.

Observe /var/log/secure and see an attempt to use password to login to target host.

May  5 11:09:33 foreman sshd[1078338]: pam_access(sshd:auth): access denied for user `foreman-remote' from `foreman.example.com'
May  5 11:09:34 foreman sshd[1078338]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=foreman.example.com user=foreman-remote
May  5 11:09:34 foreman sshd[1078338]: pam_sss(sshd:auth): received for user foreman-remote: 9 (Authentication service cannot retrieve authentication info)
May  5 11:09:36 foreman sshd[1078338]: Failed password for foreman-remote from 123.123.123.123 port 33666 ssh2
May  5 11:09:37 foreman sshd[1078338]: Postponed publickey for foreman-remote from 123.123.123.123 port 33666 ssh2 [preauth]

Running same job but setting a passphrase for the key under the advanced fields.

Same results in /var/log/secure.

May  5 11:14:53 foreman sshd[1078626]: pam_access(sshd:auth): access denied for user `foreman-remote' from `foreman.example.com'
May  5 11:14:53 foreman sshd[1078626]: pam_sss(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=foreman.example.com user=foreman-remote
May  5 11:14:53 foreman sshd[1078626]: pam_sss(sshd:auth): received for user foreman-remote: 9 (Authentication service cannot retrieve authentication info)
May  5 11:14:55 foreman sshd[1078626]: Failed password for foreman-remote from 123.123.123.123 port 34924 ssh2
May  5 11:14:56 foreman sshd[1078626]: Postponed publickey for foreman-remote from 123.123.123.123 port 34924 ssh2 [preauth]

So is there a proper way to set the password to default. empty in this case. And shouldn’t using a passhprase under the advanced options override the password login attempt to try and use the key first?

George · May 5, 2022, 8:30am

And to add to this comment. Production logs on the 11:09 run and 11:14 (passphrase)

As we can see no passphrase set apparently even though it was set earlier with the hammer command.

2022-05-05T11:09:16 [I|app|d18e7f56]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"tkp5iEvCTg7I7KFntDQI0CgAQS02X0MTWIBBV14i/ExRYohpnb6cbjayTphsJB5N6zYgog9OuOOFWKzh1ui0kA==", "job_invocation"=>{"job_category"=>"Packages", "remote_exe
cution_feature_id"=>"", "providers"=>{"SSH"=>{"job_template_id"=>"150", "job_templates"=>{"150"=>{"input_values"=>"[FILTERED]", "effective_user"=>"", "execution_timeout_interval"=>""}}}}, "description"=>"", "description_override"=>"Run %
{command}", "description_format"=>"Run %{command}", "password"=>"[FILTERED]", "key_passphrase"=>"", "effective_user_password"=>"[FILTERED]", "concurrency_level"=>"", "time_span"=>""}, "targeting"=>{"bookmark_id"=>"", "search_query"=>"nam
e ^ (foreman.example.com)", "randomized_ordering"=>"false", "targeting_type"=>"static_query"}, "fakepassword"=>"[FILTERED]", "triggering"=>{"mode"=>"immediate", "start_at_raw"=>"2022-05-05 11:09", "start_before_raw"=>"", "input_type"=>
"daily", "cronline"=>"", "days"=>"", "days_of_week"=>{"1"=>"0", "2"=>"0", "3"=>"0", "4"=>"0", "5"=>"0", "6"=>"0", "7"=>"0"}, "time"=>{"time(1i)"=>"2022", "time(2i)"=>"5", "time(3i)"=>"5", "time(4i)"=>"11", "time(5i)"=>"09"}, "max_iterati
on"=>"", "end_time_limited"=>"false", "end_time"=>{"end_time(1i)"=>"2022", "end_time(2i)"=>"5", "end_time(3i)"=>"5", "end_time(4i)"=>"11", "end_time(5i)"=>"09"}, "purpose"=>""}}

and for some reason it logged it twice?

2022-05-05T11:09:33 [I|app|7fda3b13]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"tyOrQYRDhz63JLBFAl8qnDNIKImrCalIbWHW888LhSJQC1qgUj9VXkl6X7raTzwB8H5JBpIYUriwuTtFR8HN/g==", "job_invocation"=>{"job_category"=>"Packages", "remote_exe
cution_feature_id"=>"", "providers"=>{"SSH"=>{"job_template_id"=>"143", "job_templates"=>{"141"=>{"effective_user"=>"", "execution_timeout_interval"=>""}, "143"=>{"input_values"=>"[FILTERED]", "effective_user"=>"", "execution_timeout_int
erval"=>""}}}}, "description"=>"install package(s) ", "description_override"=>"%{action} package(s) %{package}", "description_format"=>"%{action} package(s) %{package}", "password"=>"[FILTERED]", "key_passphrase"=>"", "effective_user_pas
sword"=>"[FILTERED]", "concurrency_level"=>"", "time_span"=>""}, "targeting"=>{"bookmark_id"=>"", "search_query"=>"name ^ (foreman.example.com", "randomized_ordering"=>"false", "targeting_type"=>"static_query"}, "fakepassword"=>"[FILT
ERED]", "triggering"=>{"mode"=>"immediate", "start_at_raw"=>"2022-05-05 11:09", "start_before_raw"=>"", "input_type"=>"daily", "cronline"=>"", "days"=>"", "days_of_week"=>{"1"=>"0", "2"=>"0", "3"=>"0", "4"=>"0", "5"=>"0", "6"=>"0", "7"=>
"0"}, "time"=>{"time(1i)"=>"2022", "time(2i)"=>"5", "time(3i)"=>"5", "time(4i)"=>"11", "time(5i)"=>"09"}, "max_iteration"=>"", "end_time_limited"=>"false", "end_time"=>{"end_time(1i)"=>"2022", "end_time(2i)"=>"5", "end_time(3i)"=>"5", "e
nd_time(4i)"=>"11", "end_time(5i)"=>"09"}, "purpose"=>""}, "commit"=>"Submit"}

And the key passphrase run. The key_passphrase was changed since it was plaintext. So even though the passphrase was specified it still attempted to use password on login and dismissed the key usage entirely.

2022-05-05T11:14:52 [I|app|da7250a6]   Parameters: {"utf8"=>"✓", "authenticity_token"=>"hMxydJ8FvO2okXA7E7trMaFilIrJ3hi3GqMQNCwFhoFj5IOVSXlujVbPn8TLq32sYlT1BfDP40fHe/2CpM/OXQ==", "job_invocation"=>{"job_category"=>"Packages", "remote_exe
cution_feature_id"=>"", "providers"=>{"SSH"=>{"job_template_id"=>"143", "job_templates"=>{"141"=>{"effective_user"=>"", "execution_timeout_interval"=>""}, "143"=>{"input_values"=>"[FILTERED]", "effective_user"=>"", "execution_timeout_int
erval"=>""}}}}, "description"=>"", "description_override"=>"%{action} package(s) %{package}", "description_format"=>"%{action} package(s) %{package}", "password"=>"[FILTERED]", "key_passphrase"=>"THIS_IS_PLAINTEXT", "effective_user_pa
ssword"=>"[FILTERED]", "concurrency_level"=>"", "time_span"=>""}, "targeting"=>{"bookmark_id"=>"", "search_query"=>"name ^ (foreman.example.com)", "randomized_ordering"=>"false", "targeting_type"=>"static_query"}, "fakepassword"=>"[FIL
TERED]", "triggering"=>{"mode"=>"immediate", "start_at_raw"=>"2022-05-05 11:13", "start_before_raw"=>"", "input_type"=>"daily", "cronline"=>"", "days"=>"", "days_of_week"=>{"1"=>"0", "2"=>"0", "3"=>"0", "4"=>"0", "5"=>"0", "6"=>"0", "7"=
>"0"}, "time"=>{"time(1i)"=>"2022", "time(2i)"=>"5", "time(3i)"=>"5", "time(4i)"=>"11", "time(5i)"=>"13"}, "max_iteration"=>"", "end_time_limited"=>"false", "end_time"=>{"end_time(1i)"=>"2022", "end_time(2i)"=>"5", "end_time(3i)"=>"5", "
end_time(4i)"=>"11", "end_time(5i)"=>"13"}, "purpose"=>""}, "commit"=>"Submit"}

gvde · May 5, 2022, 9:20am

Check your pam configuration. The host is set up with pam_access, i.e. it’s probably reading /etc/security/access.conf and that file doesn’t allow logins from foreman.example.com for user foreman-remote.

George · May 5, 2022, 9:46am

Like said in the starting post manually testing with the key works. No issue with pam. Issue is getting foreman to use the ssh key and the passphrase.

aruzicka · May 5, 2022, 10:00am

It probably should, but currently it does not. If both password and passphrase are provided, then the password “wins”. This is being worked on

The request shows only what was input through the form, values from settings do not get loaded into the form and then sent back so this is expected.

Those appear to be two different requests?

Apparently not. The API strictly disallows settings values to nil and UI only allows setting it to string-y valued, but even an empty string is considered a password (Bug #34867: ssh password and key passphrase should be treated as nil if they are empty strings - Foreman Remote Execution - Foreman).

As a workaround, you could probably do


echo 'Setting[:remote_execution_ssh_password] = nil' | foreman-rake console

gvde · May 5, 2022, 10:20am

If you think so, because the log is clear: pam_access denies access…

George · May 5, 2022, 11:33am

Yes thanks this worked. Also used to reset the passphrase just to be sure.

echo 'Setting[:remote_execution_ssh_key_passphrase] = nil' | foreman-rake console

Also added foreman-remote as effective user since that caused the job to be run without a user_name on the remote system.


   1:
sudo: unknown user: /home/foreman-remote/foreman-ssh-cmd-bafe4a26-d881-4e54-96bd-283a433d71a2/script
   2:
sudo: unable to initialize policy plugin
   3:
Exit status: 1

Did not need to modify access.conf to resolve this issue.

This was the issue as @aruzicka descibed.


   1:
foreman-remote
   2:
Thu May  5 14:32:08 EEST 2022
   3:
Exit status: 0

Slings · September 20, 2022, 10:03am

Hello,

i have the same problem.
I tryed the resolution without sucess.

My remote execution from foreman only takes the passwort authentication.
In the /var/log/secure i have errors like passord for remote_foreman user is false.

I want that foreman use the key authentification.

Foreman 3.3.0
foreman_remote_execution 7.1.1

How i can sett the key authentification to default ?
The remote job in the web ui is also pending status without errors.

Thanks for help.

George · October 18, 2022, 11:35am

You can try to set it to nil value. Worked for me atleast. And just to be sure you can set the passphrase to nil also.

Then proceed to add the ssh passphrase from Administer → Settings → Remote Execution → Default SSH Key Passphrase.
Also would not hurt to restart foreman services at this point

foreman-maintain services restart

Also make sure that you are using the correct ssh key. You can check it from here.
/etc/foreman-proxy/settings.d/remote_execution_ssh.yml

You can also test if the key actually works with. And then proceeding to give the ssh passphrase when prompted.

ssh -i private-ssh-key foreman-remote-execution-user@target.example.com

Hope this helps.