Remote execution not using jump host/bastion host configuration
Remote execution tasks should use the local SSH configuration to jump through bastion hosts to reach the client host
Foreman and Proxy versions:
Foreman - 3.5.1
Katello - 4.7
Foreman and Proxy plugin versions: 3.5.1
Distribution and version: Rocky Linux 8.7
Other relevant data:
- My organisation uses bastion hosts to access our suite of servers and I have added the following configuration in /etc/ssh/ssh_config.d/06-platbot.conf (so that the configuration becomes system-wide):
CheckHostIP no
ForwardAgent yes
AddKeysToAgent yes
StrictHostKeyChecking=no
HostKeyAlgorithms +ssh-dss
Ciphers +aes256-cbc,3des-cbc
KexAlgorithms +diffie-hellman-group1-sha1
IdentityFile /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy
Host *.platform.is !bastion.platform.is !<My bastion host FQDN>
ProxyJump platbot@<My bastion host FQDN>
- I have used the pub/private key for a user that we use for automation which exists on all our client hosts, and placed it in /usr/share/foreman-proxy/.ssh/:
# ls -al ~foreman-proxy/.ssh/id*
-rw-------. 1 foreman-proxy foreman-proxy 3381 Nov 30 09:01 /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy
-rw-r--r--. 1 foreman-proxy foreman-proxy 737 Nov 30 09:02 /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy.pub
-
The user under Administer → Settings → Remote Execution → SSH User has been changed to the user “platbot”
-
When executing the SSH command below which utilizes the appropriate configuration listed above, as the root user, the connection is successful:
# ssh -vv -i /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy platbot@bootstrap01-stage-bry.okd4-stage.platform.is
OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host bootstrap01-stage-bry.okd4-stage.platform.is originally bootstrap01-stage-bry.okd4-stage.platform.is
debug2: match not found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Applying options for *.platform.is
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host bootstrap01-stage-bry.okd4-stage.platform.is originally bootstrap01-stage-bry.okd4-stage.platform.is
debug2: match found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
debug2: add_identity_file: ignoring duplicate key /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy
debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Applying options for *.platform.is
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l platbot -vv -W '[%h]:%p' <My bastion host FQDN>
debug1: Executing proxy command: exec ssh -l platbot -vv -W '[bootstrap01-stage-bry.okd4-stage.platform.is]:22' <My bastion host FQDN>
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy type 0
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy-cert type -1
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy type 0
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host <My bastion host FQDN> originally <My bastion host FQDN>
debug2: match not found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Skipping Host block because of negated match for <My bastion host FQDN>
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host <My bastion host FQDN> originally <My bastion host FQDN>
debug2: match found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
debug2: add_identity_file: ignoring duplicate key /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy
debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Skipping Host block because of negated match for <My bastion host FQDN>
debug2: resolving "<My bastion host FQDN>" port 22
debug2: ssh_connect_direct
debug1: Connecting to <My bastion host FQDN> [172.21.0.2] port 22.
debug1: Connection established.
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy type 0
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2 FreeBSD-20160310
debug1: match: OpenSSH_7.2 FreeBSD-20160310 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to <My bastion host FQDN>:22 as 'platbot'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes256-cbc,3des-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:PVLEk5jJrifM1ykaiU8ryb5Wu63rnwUoJRw54BkJAS0
debug1: Host '<My bastion host FQDN>' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs
debug1: Authentication succeeded (publickey).
Authenticated to <My bastion host FQDN> ([172.21.0.2]:22).
debug1: channel_connect_stdio_fwd bootstrap01-stage-bry.okd4-stage.platform.is:22
debug2: fd 4 setting O_NONBLOCK
debug1: channel 0: new [stdio-forward]
debug2: fd 6 setting O_NONBLOCK
debug1: getpeername failed: Bad file descriptor
debug2: fd 3 setting TCP_NODELAY
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug2: fd 6 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to bootstrap01-stage-bry.okd4-stage.platform.is:22 as 'platbot'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes256-cbc,3des-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:E0Ou/LKhoMDbRMm4Kgi7DPzG6CwDUF5yXppm5rps0Xo
debug1: Host 'bootstrap01-stage-bry.okd4-stage.platform.is' is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:4
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs explicit
debug1: Will attempt key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs explicit
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs explicit
debug1: Authentication succeeded (publickey).
Authenticated to bootstrap01-stage-bry.okd4-stage.platform.is (via proxy).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: proc
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: /home/platbot/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/platbot/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LC_ALL = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug1: Sending env LANG = en_ZA.UTF-8
debug2: channel 0: request env confirm 0
debug1: Sending env LC_CTYPE = UTF-8
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Wed Feb 1 08:39:51 2023 from 172.21.0.2
[platbot@bootstrap01-stage-bry ~]$
- I can su to the foreman-proxy user and SSH to the target host, which jumps through my bastion host:
# su - foreman-proxy -c /bin/bash
bash: cannot set terminal process group (-1): Inappropriate ioctl for device
bash: no job control in this shell
bash-4.4$ id
uid=295(foreman-proxy) gid=296(foreman-proxy) groups=296(foreman-proxy) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023
bash-4.4$ ssh -vv platbot@bootstrap01-stage-bry.okd4-stage.platform.is
OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host bootstrap01-stage-bry.okd4-stage.platform.is originally bootstrap01-stage-bry.okd4-stage.platform.is
debug2: match not found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Applying options for *.platform.is
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host bootstrap01-stage-bry.okd4-stage.platform.is originally bootstrap01-stage-bry.okd4-stage.platform.is
debug2: match found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
debug2: add_identity_file: ignoring duplicate key /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy
debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Applying options for *.platform.is
debug1: Setting implicit ProxyCommand from ProxyJump: ssh -l platbot -vv -W '[%h]:%p' <My bastion host FQDN>
debug1: Executing proxy command: exec ssh -l platbot -vv -W '[bootstrap01-stage-bry.okd4-stage.platform.is]:22' <My bastion host FQDN>
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy type 0
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
OpenSSH_8.0p1, OpenSSL 1.1.1k FIPS 25 Mar 2021
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host <My bastion host FQDN> originally <My bastion host FQDN>
debug2: match not found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Skipping Host block because of negated match for <My bastion host FQDN>
debug1: configuration requests final Match pass
debug1: re-parsing configuration
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug2: checking match for 'final all' host <My bastion host FQDN> originally <My bastion host FQDN>
debug2: match found
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
debug2: add_identity_file: ignoring duplicate key /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy
debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Skipping Host block because of negated match for <My bastion host FQDN>
debug2: resolving "<My bastion host FQDN>" port 22
debug2: ssh_connect_direct
debug1: Connecting to <My bastion host FQDN> [172.21.0.2] port 22.
debug1: Connection established.
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy type 0
debug1: identity file /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.0
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2 FreeBSD-20160310
debug1: match: OpenSSH_7.2 FreeBSD-20160310 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to <My bastion host FQDN>:22 as 'platbot'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes256-cbc,3des-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp521,ecdh-sha2-nistp384,ecdh-sha2-nistp256,diffie-hellman-group-exchange-sha256
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ssh-dss,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes256-gcm@openssh.com,aes128-gcm@openssh.com,aes256-ctr,aes192-ctr,aes128-ctr
debug2: MACs ctos: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: MACs stoc: hmac-sha2-512-etm@openssh.com,hmac-sha2-256-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512,hmac-sha2-256,umac-128@openssh.com
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
debug1: kex: curve25519-sha256@libssh.org need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:PVLEk5jJrifM1ykaiU8ryb5Wu63rnwUoJRw54BkJAS0
debug1: Host '<My bastion host FQDN>' is known and matches the ECDSA host key.
debug1: Found key in /usr/share/foreman-proxy/.ssh/known_hosts:2
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs
debug1: Authentication succeeded (publickey).
Authenticated to <My bastion host FQDN> ([172.21.0.2]:22).
debug1: channel_connect_stdio_fwd bootstrap01-stage-bry.okd4-stage.platform.is:22
debug2: fd 4 setting O_NONBLOCK
debug1: channel 0: new [stdio-forward]
debug2: fd 6 setting O_NONBLOCK
debug1: getpeername failed: Bad file descriptor
debug2: fd 3 setting TCP_NODELAY
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug1: Remote protocol version 2.0, remote software version OpenSSH_8.0
debug1: match: OpenSSH_8.0 pat OpenSSH* compat 0x04000000
debug2: fd 6 setting O_NONBLOCK
debug2: fd 4 setting O_NONBLOCK
debug1: Authenticating to bootstrap01-stage-bry.okd4-stage.platform.is:22 as 'platbot'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes256-cbc,3des-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: ciphers stoc: aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes256-ctr,aes256-cbc,aes128-gcm@openssh.com,aes128-ctr,aes128-cbc
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha2-256,hmac-sha1,umac-128@openssh.com,hmac-sha2-512
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: kex: curve25519-sha256 need=64 dh_need=64
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:E0Ou/LKhoMDbRMm4Kgi7DPzG6CwDUF5yXppm5rps0Xo
debug1: Host 'bootstrap01-stage-bry.okd4-stage.platform.is' is known and matches the ECDSA host key.
debug1: Found key in /usr/share/foreman-proxy/.ssh/known_hosts:4
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: /usr/share/foreman-proxy/.ssh/id_rsa_foreman_proxy RSA SHA256:oHEDJ+/Zs1x/YYU7YE9s1lxt9Crhniyf/htec1ukCLs
debug1: Authentication succeeded (publickey).
Authenticated to bootstrap01-stage-bry.okd4-stage.platform.is (via proxy).
debug1: channel 0: new [client-session]
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: proc
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Remote: /home/platbot/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug1: Remote: /home/platbot/.ssh/authorized_keys:2: key options: agent-forwarding port-forwarding pty user-rc x11-forwarding
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: client_session2_setup: id 0
debug2: channel 0: request pty-req confirm 1
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug2: channel 0: request shell confirm 1
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 0 rmax 32768
debug2: channel_input_status_confirm: type 99 id 0
debug2: PTY allocation request accepted on channel 0
debug2: channel 0: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
Last login: Wed Feb 1 09:22:29 2023 from 172.21.0.2
[platbot@bootstrap01-stage-bry ~]$
-
I first attempted to execute a simple command via the GUI (Hosts → All Hosts → Selected the tickbox for the host → Select Action → Schedule Remote Job, against a client host in the same IP range as the master host. While this should still use the SSH bastion config, it will still work if it doesn’t as it is in the same range, and it works. So I assume that the remote execution plugin and the SSH key’s and SSH user are all good.
-
However, when attempting to execute a job against the same host in the snippets above (the access for which worked manually using the system-wide configuration), the job times out.
-
The debug level settings were increased in verbosity in the following files:
# grep -i debug /etc/foreman-proxy/settings.yml
# WARN, DEBUG, ERROR, FATAL, INFO, UNKNOWN
:log_level: DEBUG
# grep -i debug /etc/foreman-proxy/settings.d/remote_execution_ssh.yml
:ssh_log_level: debug
When rerunning the job with these increased debug levels, the output in /var/log/foreman-proxy is as follows:
2023-02-01T09:20:06 [D] Executor heartbeat
2023-02-01T09:20:19 [D] accept: 172.20.9.26:56414
2023-02-01T09:20:19 [D] Rack::Handler::WEBrick is invoked.
2023-02-01T09:20:19 b70782e9 [I] Started GET /dynflow/tasks/count state=running
2023-02-01T09:20:19 b70782e9 [D] verifying remote client 172.20.9.26 against trusted_hosts ["forekat-master01-stage-bry.platform.is"]
2023-02-01T09:20:19 b70782e9 [I] Finished GET /dynflow/tasks/count with 200 (1.03 ms)
2023-02-01T09:20:19 [D] close: 172.20.9.26:56414
2023-02-01T09:20:19 [D] accept: 172.20.9.26:56430
2023-02-01T09:20:19 [D] Rack::Handler::WEBrick is invoked.
2023-02-01T09:20:19 b70782e9 [I] Started POST /dynflow/tasks/launch
2023-02-01T09:20:19 b70782e9 [D] verifying remote client 172.20.9.26 against trusted_hosts ["forekat-master01-stage-bry.platform.is"]
2023-02-01T09:20:19 b70782e9 [D] ExecutionPlan e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c pending >> planning
2023-02-01T09:20:19 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 1 pending >> running in phase Plan Proxy::Dynflow::Action::Batch
2023-02-01T09:20:19 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 1 running >> success in phase Plan Proxy::Dynflow::Action::Batch
2023-02-01T09:20:19 b70782e9 [D] ExecutionPlan e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c planning >> planned
2023-02-01T09:20:19 [D] ExecutionPlan e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c planned >> running
2023-02-01T09:20:19 b70782e9 [I] Finished POST /dynflow/tasks/launch with 200 (21.25 ms)
2023-02-01T09:20:19 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 pending >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:20:19 b70782e9 [D] ExecutionPlan 0b62997c-57f4-45d2-9204-f0ce70c5e6ec pending >> planning
2023-02-01T09:20:19 b70782e9 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 1 pending >> running in phase Plan Proxy::RemoteExecution::Ssh::Actions::RunScript
2023-02-01T09:20:19 b70782e9 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 2 pending >> running in phase Plan Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2023-02-01T09:20:19 b70782e9 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 5 pending >> running in phase Plan Proxy::Dynflow::Callback::Action
2023-02-01T09:20:19 b70782e9 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 5 running >> success in phase Plan Proxy::Dynflow::Callback::Action
2023-02-01T09:20:19 b70782e9 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 2 running >> success in phase Plan Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2023-02-01T09:20:19 b70782e9 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 1 running >> success in phase Plan Proxy::RemoteExecution::Ssh::Actions::RunScript
2023-02-01T09:20:19 b70782e9 [D] ExecutionPlan 0b62997c-57f4-45d2-9204-f0ce70c5e6ec planning >> planned
2023-02-01T09:20:19 [D] close: 172.20.9.26:56430
2023-02-01T09:20:19 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:20:19 [D] ExecutionPlan 0b62997c-57f4-45d2-9204-f0ce70c5e6ec planned >> running
2023-02-01T09:20:19 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 3 pending >> running in phase Run Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2023-02-01T09:20:19 b70782e9 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 3 running >> suspended in phase Run Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2023-02-01T09:20:19 [D] start runner d80ceb13-779f-4aee-baed-6f403fcf71a3
2023-02-01T09:20:19 [D] Checking if private key has passphrase: ssh-keygen -y -f /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
2023-02-01T09:20:19 [D] Private key is not protected with a passphrase
2023-02-01T09:20:19 [D] Running: ssh -o User=platbot -o Port=22 -o IdentityFile=/var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=debug -o ControlMaster=auto -o ControlPath=/var/tmp/d80ceb13-779f-4aee-baed-6f403fcf71a3 -o ControlPersist=yes -o ProxyCommand=none -o PreferredAuthentications=publickey -o NumberOfPasswordPrompts=0 bootstrap01-stage-bry.okd4-stage.platform.is true
2023-02-01T09:20:19 [D] debug1: Reading configuration data /etc/ssh/ssh_config
2023-02-01T09:20:19 [D] debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
2023-02-01T09:20:19 [D] debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
2023-02-01T09:20:19 [D] debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
2023-02-01T09:20:19 [D] debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Applying options for *.platform.is
2023-02-01T09:20:19 [D] debug1: configuration requests final Match pass
2023-02-01T09:20:19 [D] debug1: re-parsing configuration
2023-02-01T09:20:19 [D] debug1: Reading configuration data /etc/ssh/ssh_config
2023-02-01T09:20:19 [D] debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
2023-02-01T09:20:19 [D] debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
2023-02-01T09:20:19 [D] debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
2023-02-01T09:20:19 [D] debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Applying options for *.platform.is
2023-02-01T09:20:19 [D] debug1: auto-mux: Trying existing master
2023-02-01T09:20:19 [D] debug1: Control socket "/var/tmp/d80ceb13-779f-4aee-baed-6f403fcf71a3" does not exist
2023-02-01T09:20:19 [D] debug1: Connecting to bootstrap01-stage-bry.okd4-stage.platform.is [172.20.8.253] port 22.
2023-02-01T09:20:21 [D] Executor heartbeat
2023-02-01T09:20:29 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:20:29 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:20:29 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:20:36 [D] Executor heartbeat
2023-02-01T09:20:39 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:20:39 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:20:39 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:20:49 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:20:49 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:20:49 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:20:51 [D] Executor heartbeat
2023-02-01T09:20:59 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:20:59 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:20:59 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:06 [D] Executor heartbeat
2023-02-01T09:21:09 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:21:09 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:09 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:19 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:21:19 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:19 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:21 [D] Executor heartbeat
2023-02-01T09:21:29 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:21:29 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:29 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:36 [D] Executor heartbeat
2023-02-01T09:21:39 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:21:39 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:39 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:49 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:21:49 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:49 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:51 [D] Executor heartbeat
2023-02-01T09:21:59 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:21:59 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:21:59 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:22:06 [D] Executor heartbeat
2023-02-01T09:22:09 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:22:09 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:22:09 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:22:19 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:22:19 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:22:19 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:22:21 [D] Executor heartbeat
2023-02-01T09:22:29 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:22:29 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:22:29 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> suspended in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:22:31 [D] debug1: connect to address 172.20.8.253 port 22: Connection timed out
2023-02-01T09:22:31 [D] ssh: connect to host bootstrap01-stage-bry.okd4-stage.platform.is port 22: Connection timed out
2023-02-01T09:22:31 [D] Failed to establish connection using authentication method publickey
2023-02-01T09:22:31 [E] error while initializing command RuntimeError Could not establish connection to remote host using any available authentication method, tried publickey:
/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.9.0/lib/smart_proxy_remote_execution_ssh/multiplexed_ssh_connection.rb:75:in `establish!'
/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.9.0/lib/smart_proxy_remote_execution_ssh/runners/script_runner.rb:149:in `start'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/runner/dispatcher.rb:32:in `start_runner'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/actor.rb:13:in `on_message'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/runner/dispatcher.rb:24:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/actor.rb:122:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/actor.rb:56:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-02-01T09:22:31 [E] Error initializing command - RuntimeError Could not establish connection to remote host using any available authentication method, tried publickey:
/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.9.0/lib/smart_proxy_remote_execution_ssh/multiplexed_ssh_connection.rb:75:in `establish!'
/usr/share/gems/gems/smart_proxy_remote_execution_ssh-0.9.0/lib/smart_proxy_remote_execution_ssh/runners/script_runner.rb:149:in `start'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/runner/dispatcher.rb:32:in `start_runner'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/actor.rb:13:in `on_message'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/usr/share/gems/gems/smart_proxy_dynflow-0.9.0/lib/smart_proxy_dynflow/runner/dispatcher.rb:24:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/actor.rb:122:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/dynflow-1.6.8/lib/dynflow/actor.rb:56:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `block in synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:47:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:352:in `run_task'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:343:in `block (3 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `loop'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:334:in `block (2 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `catch'
/usr/share/gems/gems/concurrent-ruby-1.1.10/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:333:in `block in create_worker'
/usr/share/gems/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-02-01T09:22:31 [D] refresh runner d80ceb13-779f-4aee-baed-6f403fcf71a3
2023-02-01T09:22:31 [D] refreshing runner
2023-02-01T09:22:31 [D] finish runner d80ceb13-779f-4aee-baed-6f403fcf71a3
2023-02-01T09:22:31 [D] closing session for command [d80ceb13-779f-4aee-baed-6f403fcf71a3],0 actors left
2023-02-01T09:22:31 [D] terminate d80ceb13-779f-4aee-baed-6f403fcf71a3
2023-02-01T09:22:31 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 3 got event #<Proxy::Dynflow::Runner::Update:0x000055de1c111908>
2023-02-01T09:22:31 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 3 suspended >> running in phase Run Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2023-02-01T09:22:31 b70782e9 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 3 running >> success in phase Run Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2023-02-01T09:22:31 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 6 pending >> running in phase Run Proxy::Dynflow::Callback::Action
2023-02-01T09:22:31 b70782e9 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 6 running >> success in phase Run Proxy::Dynflow::Callback::Action
2023-02-01T09:22:31 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 4 pending >> running in phase Finalize Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2023-02-01T09:22:31 b70782e9 [E] Script execution failed
2023-02-01T09:22:31 b70782e9 [D] Step 0b62997c-57f4-45d2-9204-f0ce70c5e6ec: 4 running >> error in phase Finalize Proxy::RemoteExecution::Ssh::Actions::ScriptRunner
2023-02-01T09:22:31 [D] ExecutionPlan 0b62997c-57f4-45d2-9204-f0ce70c5e6ec running >> stopped
2023-02-01T09:22:36 [D] Executor heartbeat
2023-02-01T09:22:39 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 got event Dynflow::Action::WithPollingSubPlans::Poll
2023-02-01T09:22:39 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 suspended >> running in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:22:39 b70782e9 [E] <Dynflow::Action::WithSubPlans::SubtaskFailedException> A sub task failed
2023-02-01T09:22:39 b70782e9 [D] Step e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c: 2 running >> error in phase Run Proxy::Dynflow::Action::Batch
2023-02-01T09:22:39 [D] ExecutionPlan e1f3b9f6-ea5c-4c47-95a0-94d79f2d171c running >> stopped
- The job appears to time out due to the destination host being unreachable, due to the connection not jumping through the bastion host. I does appear to be parsing the SSH configuration file:
2023-02-01T09:20:19 [D] debug1: Reading configuration data /etc/ssh/ssh_config.d/06-platbot.conf
2023-02-01T09:20:19 [D] debug1: /etc/ssh/ssh_config.d/06-platbot.conf line 11: Applying options for *.platform.is
- I also noticed the ProxyCommand=none portion of this line:
2023-02-01T09:20:19 [D] Running: ssh -o User=platbot -o Port=22 -o IdentityFile=/var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o LogLevel=debug -o ControlMaster=auto -o ControlPath=/var/tmp/d80ceb13-779f-4aee-baed-6f403fcf71a3 -o ControlPersist=yes -o ProxyCommand=none -o PreferredAuthentications=publickey -o NumberOfPasswordPrompts=0 bootstrap01-stage-bry.okd4-stage.platform.is true
I feel like I’m being a muppet and missing something basic here and any assistance would be appreciated!