Forman-Puppet-Windows Nodes - Unable to create remote directory

mdyunusraza · November 21, 2020, 5:08am

Problem:
Hi Community,

I am a first time user of Foreman & Puppet and recently started this PoC to see if this could be beneficial to our business. So to begin on Centos 8 , I installed my latest opensource Puppet master (6.19) and then on top of that configured the latest Foreman. After this I installed the Puppet agents on 1 linux node and 1 Windows 10 node. The Windows 10 node was also installed with OpenSSH to expose SSH. I tested manuallly and I can do SSH to Windows 10 with username/password.

In Foreman its reporting facts and everything looks fine from server-client communication.

The problem is when i am trying to do a remote job on Windows 10 its failing with the error below. I can see that its trying to create a /var/tmp on a windows machine and obviously thats not possible. I would like to know if Foreman can manage Windows nodes using Puppet. I have not tried with Ansible but I am sure it will have same issues.

1:
Error initializing command: RuntimeError - Unable to create directory on remote system /var/tmp/foreman-ssh-cmd-9f354046-326e-4c6a-be32-db79894136ef: exit code: 1
2:
The syntax of the command is incorrect.
3:
Exit status: EXCEPTION

Expected outcome:
I should be able to run a remote job using Puppet

Foreman and Proxy versions:
version 2.2.1
Foreman and Proxy plugin versions:

Distribution and version:
CentOS 8
Windows 10

Other relevant data:

mdyunusraza · November 26, 2020, 5:18am

Anyone??

ekohl · December 2, 2020, 11:49am

When you say remote job,you mean remote execution from Foreman, right?

@aruzicka it looks like the temporary directory assumes a Linux directory structure for the temporary directory. Any advice? Would Ansible work with this?

aruzicka · December 2, 2020, 1:35pm

The remote path is configurable and defaults to /var/tmp. Maybe if you changed to it some windows path it could work, but that’s just a wild guess.

Ansible could work, but ymmv

mdyunusraza · December 2, 2020, 3:35pm

@ekohl yes remote execution via foreman-proxy

@aruzicka where can i change that? I also installed cygwin on windows to get linux directory structure but its the same thing if i manually run scp -v script user@windows /var/tmp then i can see file is uploaded but foreman is not able to do same

aruzicka · December 2, 2020, 3:47pm

It should be described here

mdyunusraza · December 3, 2020, 8:59am

I changed that and now i can see that a folder got created “C:\Temp\foreman-ssh-cmd-b94c8218-8e5d-421e-a948-d87ca8e8b060” however i have the error on foreman-proxy task

1:
Error initializing command: RuntimeError - Unable to upload file to C:/Temp/foreman-ssh-cmd-b94c8218-8e5d-421e-a948-d87ca8e8b060/script on remote system: exit code: 1
2:
Exit status: EXCEPTION

mdyunusraza · December 9, 2020, 9:08am

@aruzicka, any idea why it would not be able to upload the file when its able to connect and create a folder on the remote host?

aruzicka · December 9, 2020, 9:43am

Not really, but looking at the path it looks like it gets “reconstructed” the non-windows way since the path gets built on a linux box before opening a connection to the other side.

Have you tried ansible?

mdyunusraza · December 9, 2020, 10:04am

No i have not tried ansible as that is not an option for me. But what i am not able to understand that if i run the command manually from the foreman-proxy it is able to upload the “script” file but what exactly happens when foreman does from gui is not clear to me.

aruzicka · December 9, 2020, 10:17am

As in ssh $host $command?

The script uploading part roughly translates to echo $SCRIPT | ssh $HOST sh -c "cat > C:/Temp/foreman-ssh-cmd-b94c8218-8e5d-421e-a948-d87ca8e8b060/script"

mdyunusraza · December 9, 2020, 10:24am

From foreman-proxy i run this command:
scp -v script administrator@server01a:C:/Temp

This is able to upload the script file to server01a in C:\Temp directory

mdyunusraza · December 9, 2020, 10:38am

scp -v script administrator@server01a:/var/tmp/
Executing: program /usr/bin/ssh host server01a, user administrator, command scp -v -t /var/tmp/
OpenSSH_7.8p1, OpenSSL 1.1.1c FIPS 28 May 2019
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Reading configuration data /etc/ssh/ssh_config.d/05-redhat.conf
debug1: Reading configuration data /etc/crypto-policies/back-ends/openssh.config
debug1: /etc/ssh/ssh_config.d/05-redhat.conf line 8: Applying options for *
debug1: Connecting to server01a [192.168.1.35] port 22.
debug1: Connection established.
debug1: identity file /root/.ssh/id_rsa type 0
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: identity file /root/.ssh/id_xmss type -1
debug1: identity file /root/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_for_Windows _7.7
debug1: match: OpenSSH_for_Windows_7.7 pat OpenSSH* compat 0x04000000
debug1: Authenticating to server01a:22 as ‘administrator’
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: aes256-gcm@openssh.com MAC: compr ession: none
debug1: kex: client->server cipher: aes256-gcm@openssh.com MAC: compr ession: none
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: kex: curve25519-sha256@libssh.org need=32 dh_need=32
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:G8mYYho+XihmAeAcWbdsCWGcfiDG RoZuccj1AET3+/w
debug1: Host ‘server01a’ is known and matches the ECDSA host key.
debug1: Found key in /root/.ssh/known_hosts:1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rs a-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Next authentication method: publickey
debug1: Offering public key: RSA SHA256:s68dg5SBrLJcPweuq1hxXkIElPjDdOmZAEYkoXfe frA /root/.ssh/id_rsa
debug1: Server accepts key: pkalg rsa-sha2-256 blen 279
debug1: Authentication succeeded (publickey).
Authenticated to server01a ([192.168.1.35]:22).
debug1: channel 0: new [client-session]
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.
debug1: pledge: network
debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
debug1: Sending environment.
debug1: Sending env LANG = en_US.UTF-8
debug1: Sending command: scp -v -t /var/tmp/
Sending file modes: C0644 16 script
script 100% 16 0.8KB/s 00:00
Sink: C0644 16 script
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug1: channel 0: free: client-session, nchannels 1
debug1: fd 0 clearing O_NONBLOCK
Transferred: sent 2884, received 2672 bytes, in 1.1 seconds
Bytes per second: sent 2649.0, received 2454.3
debug1: Exit status 0

aruzicka · December 9, 2020, 10:41am

As I mentioned here, we are not using scp. We open the regular ssh connection, spawn the shell on the other side, feed the script as input on the local side and write it into a file on the remote side. It isn’t really the same thing

mdyunusraza · December 9, 2020, 10:55am

ssh also works manually. And when i run the job i can see that foreman creates a folder in C:\Temp but why its not able to upload the file named “script” to that folder is not clear. I looked at the dynflow and production log but it does not give me any information why its not able to upload the file .

mdyunusraza · December 9, 2020, 11:16am

ssh administrator@server01a sh -c “cat > C:/Temp/script”
fffffffff
^Z
[1]+ Stopped ssh administrator@server01a sh -c “cat > C:/Temp/script”

this created a file script in C:\Temp and had the contents as “fffff”

aruzicka · December 9, 2020, 11:40am

Interesting. Could you try bumping log level to debug in /etc/smart_proxy_dynflow_core/settings.yml and ssh log level to debug in /etc/foreman-proxy/settings.d/remote_execution_ssh.yml, restarting smart_proxy_dynflow_core service and watching /var/log/foreman-proxy/smart_proxy_dynflow_core.log while you run a job?

mdyunusraza · December 9, 2020, 12:51pm

Sure i will try tomorrow and post results. Thanks for your time

mdyunusraza · December 10, 2020, 6:28am

@aruzicka , I enabled the DEBUG logs on settings.yml and below is the output. For the SSH Debug log, i dont have the debug log entry. The file only has the following

=============

:enabled: https
:ssh_identity_key_file: /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy
:local_working_dir: /var/tmp
:remote_working_dir: C:/Temp
:kerberos_auth: false

**# Whether to run remote execution jobs asynchronously**
:async_ssh: false

–============

voked.192.168.1.76 - - [10/Dec/2020:14:19:18 +08] "GET /tasks/count?state=running HTTP/1.1" 200 29
close: 192.168.1.76:56418accept: 192.168.1.76:56424Rack::Handler::WEBrick is invoked.ExecutionPlan 65118512-2ce2-4994-a8cb-3a51f88b3d3e      pending >>  planning         Step 65118512-2ce2-4994-a8cb-3a51f88b3d3e: 1   pending >>   running in phase     Plan ForemanTasksCore::BatchActionExecutionPlan ea31c579-9235-42b1-8586-b0fc0ead5a99      pending >>  planning         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 1   pending >>   running in phase     Plan ForemanRemoteExecutionCore::Actions::RunScript         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 4   pending >>   running in phase     Plan SmartProxyDynflowCore::Callback::Action         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 4   running >>   success in phase     Plan SmartProxyDynflowCore::Callback::Action         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 1   running >>   success in phase     Plan ForemanRemoteExecutionCore::Actions::RunScriptExecutionPlan ea31c579-9235-42b1-8586-b0fc0ead5a99     planning >>   plannedExecutionPlan ea31c579-9235-42b1-8586-b0fc0ead5a99      planned >>   running         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 2   pending >>   running in phase      Run ForemanRemoteExecutionCore::Actions::RunScript         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 2   running >> suspended in phase      Run ForemanRemoteExecutionCore::Actions::RunScript         Step 65118512-2ce2-4994-a8cb-3a51f88b3d3e: 1   running >>   success in phase     Plan ForemanTasksCore::BatchActionExecutionPlan 65118512-2ce2-4994-a8cb-3a51f88b3d3e     planning >>   plannedstart runner ce218673-cace-4d42-96ce-eabab25c05f5copying script to C:/Temp/foreman-ssh-cmd-ce218673-cace-4d42-96ce-eabab25c05f5/script:
  | ipconfigopening session to administrator@192.168.1.35establishing connection to 192.168.1.35:22192.168.1.76 - - [10/Dec/2020:14:19:18 +08] "POST /tasks/launch? HTTP/1.1" 200 110
close: 192.168.1.76:56424connection establishednegotiating protocol versionlocal is `SSH-2.0-Ruby/Net::SSH_4.2.0 x86_64-linux'ExecutionPlan 65118512-2ce2-4994-a8cb-3a51f88b3d3e      planned >>   running         Step 65118512-2ce2-4994-a8cb-3a51f88b3d3e: 2   pending >>   running in phase      Run ForemanTasksCore::BatchAction         Step 65118512-2ce2-4994-a8cb-3a51f88b3d3e: 2   running >> suspended in phase      Run ForemanTasksCore::BatchActionremote is `SSH-2.0-OpenSSH_for_Windows_7.7'unsupported algorithm: `["aes256-gcm@openssh.com", "chacha20-poly1305@openssh.com", "aes128-gcm@openssh.com"]'unsupported algorithm: `["hmac-sha2-256-etm@openssh.com", "hmac-sha1-etm@openssh.com", "umac-128-etm@openssh.com", "hmac-sha2-512-etm@openssh.com", "umac-128@openssh.com"]'sending KEXINITqueueing packet nr 0 type 20 len 596sent 600 bytesread 1048 bytesreceived packet nr 0 type 20 len 1044got KEXINIT from servernegotiating algorithmsnegotiated:
* kex: diffie-hellman-group14-sha1
* host_key: ssh-rsa
* encryption_server: aes256-ctr
* encryption_client: aes256-ctr
* hmac_client: hmac-sha2-256
* hmac_server: hmac-sha2-256
* compression_client: none
* compression_server: none
* language_client:
* language_server: exchanging keysqueueing packet nr 1 type 30 len 268sent 272 bytesread 848 bytesreceived packet nr 1 type 31 len 828queueing packet nr 2 type 21 len 20sent 24 bytesreceived packet nr 2 type 21 len 12beginning authentication of `administrator'queueing packet nr 3 type 5 len 28sent 64 bytesread 64 bytesreceived packet nr 3 type 6 len 28trying passwordqueueing packet nr 4 type 50 len 60sent 96 bytesread 80 bytesreceived packet nr 4 type 51 len 44allowed methods: publickey,keyboard-interactivepassword failedtrying publickeyconnecting to ssh-agentcould not connect to ssh-agent: Agent not configuredtrying publickey (68:92:15:97:fa:07:13:9a:30:f1:26:56:4b:2a:6f:4c)queueing packet nr 5 type 50 len 364sent 400 bytesread 336 bytesreceived packet nr 5 type 60 len 300queueing packet nr 6 type 50 len 636sent 672 bytesread 48 bytesreceived packet nr 6 type 52 len 12publickey succeeded (68:92:15:97:fa:07:13:9a:30:f1:26:56:4b:2a:6f:4c)queueing packet nr 7 type 90 len 44sent 80 bytesread 528 bytesreceived packet nr 7 type 80 len 492global request received: hostkeys-00@openssh.com falseread 64 bytesreceived packet nr 8 type 91 len 28channel_open_confirmation: 0 0 0 32768sending channel request "env"queueing packet nr 8 type 98 len 44sending channel request "exec"queueing packet nr 9 type 98 len 92sent 208 bytesread 112 bytesreceived packet nr 9 type 93 len 28channel_window_adjust: 0 +2097152received packet nr 10 type 99 len 12channel_success: 0read 64 bytesreceived packet nr 11 type 94 len 28channel_data: 0 2bread 336 bytesreceived packet nr 12 type 94 len 28channel_data: 0 2breceived packet nr 13 type 94 len 44channel_data: 0 22breceived packet nr 14 type 94 len 28channel_data: 0 2breceived packet nr 15 type 94 len 28channel_data: 0 2breceived packet nr 16 type 94 len 28channel_data: 0 2bread 720 bytesreceived packet nr 17 type 94 len 140channel_data: 0 119breceived packet nr 18 type 94 len 28channel_data: 0 2breceived packet nr 19 type 94 len 140channel_data: 0 119breceived packet nr 20 type 94 len 28channel_data: 0 2breceived packet nr 21 type 94 len 140channel_data: 0 119breceived packet nr 22 type 94 len 28channel_data: 0 2bread 128 bytesreceived packet nr 23 type 94 len 28channel_data: 0 2breceived packet nr 24 type 94 len 28channel_data: 0 2bread 48 bytesreceived packet nr 25 type 96 len 12channel_eof: 0read 128 bytesreceived packet nr 26 type 98 len 44channel_request: 0 exit-status falsereceived packet nr 27 type 97 len 12channel_close: 0queueing packet nr 10 type 97 len 28192.168.1.35 delete channel 0 which closed locally and remotelysent 64 bytesSending data to C:/Temp/foreman-ssh-cmd-ce218673-cace-4d42-96ce-eabab25c05f5/script on remote host:
ipconfigqueueing packet nr 11 type 90 len 44sent 80 bytesread 64 bytesreceived packet nr 28 type 91 len 28channel_open_confirmation: 1 0 0 32768sending channel request "env"queueing packet nr 12 type 98 len 44sending channel request "exec"queueing packet nr 13 type 98 len 204sent 320 bytesread 112 bytesreceived packet nr 29 type 93 len 28channel_window_adjust: 1 +2097152received packet nr 30 type 99 len 12channel_success: 1queueing packet nr 14 type 94 len 28queueing packet nr 15 type 97 len 28sent 128 bytesread 128 bytesreceived packet nr 31 type 98 len 44channel_request: 1 exit-status falsereceived packet nr 32 type 97 len 12channel_close: 1192.168.1.35 delete channel 1 which closed locally and remotelyerror while initalizing command RuntimeError Unable to upload file to C:/Temp/foreman-ssh-cmd-ce218673-cace-4d42-96ce-eabab25c05f5/script on remote system: exit code: 1:
 /usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:422:in `upload_data'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:407:in `cp_script_to_remote'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:172:in `prepare_start'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:158:in `start'
/usr/share/gems/gems/foreman-tasks-core-0.3.4/lib/foreman_tasks_core/runner/dispatcher.rb:32:in `start_runner'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:7:in `on_message'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/usr/share/gems/gems/foreman-tasks-core-0.3.4/lib/foreman_tasks_core/runner/dispatcher.rb:24:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:106:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `block in on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:59:in `with_backtrace'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `block in synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:353:in `run_task'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:342:in `block (3 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `loop'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `block (2 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `catch'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `block in create_worker'Error initializing command - RuntimeError Unable to upload file to C:/Temp/foreman-ssh-cmd-ce218673-cace-4d42-96ce-eabab25c05f5/script on remote system: exit code: 1:
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:422:in `upload_data'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:407:in `cp_script_to_remote'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:172:in `prepare_start'
/usr/share/gems/gems/foreman_remote_execution_core-1.3.1/lib/foreman_remote_execution_core/script_runner.rb:158:in `start'
/usr/share/gems/gems/foreman-tasks-core-0.3.4/lib/foreman_tasks_core/runner/dispatcher.rb:32:in `start_runner'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:7:in `on_message'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/usr/share/gems/gems/foreman-tasks-core-0.3.4/lib/foreman_tasks_core/runner/dispatcher.rb:24:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:106:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `block in on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:59:in `with_backtrace'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `block in synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:353:in `run_task'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:342:in `block (3 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `loop'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `block (2 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `catch'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `block in create_worker'refresh runner ce218673-cace-4d42-96ce-eabab25c05f5refreshing runnerfinish runner ce218673-cace-4d42-96ce-eabab25c05f5closing session for command [ce218673-cace-4d42-96ce-eabab25c05f5],0 actors left terminate ce218673-cace-4d42-96ce-eabab25c05f5queueing packet nr 16 type 90 len 44         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 2 got event #<ForemanTasksCore::Runner::Update:0x00005649006228a8>         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 2 suspended >>   running in phase      Run ForemanRemoteExecutionCore::Actions::RunScript         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 2   running >>   success in phase      Run ForemanRemoteExecutionCore::Actions::RunScriptsent 80 bytesread 64 bytesreceived packet nr 33 type 91 len 28channel_open_confirmation: 2 0 0 32768sending channel request "env"queueing packet nr 17 type 98 len 44sending channel request "exec"queueing packet nr 18 type 98 len 92sent 208 bytes         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 5   pending >>   running in phase      Run SmartProxyDynflowCore::Callback::Actionread 112 bytesreceived packet nr 34 type 93 len 28channel_window_adjust: 2 +2097152received packet nr 35 type 99 len 12channel_success: 2         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 5   running >>   success in phase      Run SmartProxyDynflowCore::Callback::Action         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 3   pending >>   running in phase Finalize ForemanRemoteExecutionCore::Actions::RunScriptScript execution failed         Step ea31c579-9235-42b1-8586-b0fc0ead5a99: 3   running >>     error in phase Finalize ForemanRemoteExecutionCore::Actions::RunScriptExecutionPlan ea31c579-9235-42b1-8586-b0fc0ead5a99      running >>   stoppedread 224 bytesreceived packet nr 36 type 95 len 108channel_extended_data: 2 1 77breceived packet nr 37 type 95 len 44channel_extended_data: 2 1 18bread 400 bytesreceived packet nr 38 type 95 len 204channel_extended_data: 2 1 174breceived packet nr 39 type 95 len 124channel_extended_data: 2 1 105bread 48 bytesreceived packet nr 40 type 96 len 12channel_eof: 2read 128 bytesreceived packet nr 41 type 98 len 44channel_request: 2 exit-status falsereceived packet nr 42 type 97 len 12channel_close: 2queueing packet nr 19 type 97 len 28192.168.1.35 delete channel 2 which closed locally and remotelysent 64 bytesclosing remaining channels (0 open)         Step 65118512-2ce2-4994-a8cb-3a51f88b3d3e: 2 got event Dynflow::Action::WithPollingSubPlans::Poll         Step 65118512-2ce2-4994-a8cb-3a51f88b3d3e: 2 suspended >>   running in phase      Run ForemanTasksCore::BatchActionA sub task failed (RuntimeError)
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:231:in `check_for_errors!'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_sub_plans.rb:137:in `try_to_finish'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_polling_sub_plans.rb:19:in `poll'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/with_polling_sub_plans.rb:11:in `run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/parallel/worker.rb:15:in `block in on_message'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/parallel/worker.rb:14:in `on_message'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/context.rb:46:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:106:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `block in on_envelope'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:59:in `with_backtrace'
/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/actor.rb:47:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/termination.rb:55:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/abstract.rb:25:in `pass'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:162:in `process_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:96:in `block in on_envelope'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:119:in `block (2 levels) in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `block in synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/synchronization/mutex_lockable_object.rb:41:in `synchronize'
/usr/share/gems/gems/concurrent-ruby-edge-0.6.0/lib/concurrent-ruby-edge/concurrent/actor/core.rb:116:in `block in schedule_execution'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:18:in `call'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:96:in `work'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/serialized_execution.rb:77:in `block in call_job'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:353:in `run_task'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:342:in `block (3 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `loop'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:325:in `block (2 levels) in create_worker'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `catch'
/usr/share/gems/gems/concurrent-ruby-1.1.6/lib/concurrent-ruby/concurrent/executor/ruby_thread_pool_executor.rb:324:in `block in create_worker'         Step 65118512-2ce2-4994-a8cb-3a51f88b3d3e: 2   running >>     error in phase      Run ForemanTasksCore::BatchActionExecutionPlan 65118512-2ce2-4994-a8cb-3a51f88b3d3e      running >>   stoppedExecutor heartbeat^C

mdyunusraza · December 10, 2020, 6:42am

Could it be related to this bug 1451675 – Using effective user leads to Error initializing command: RuntimeError - Unable to upload file on remote system: exit code: 127 (redhat.com)

It seems it was solved for EL7 and I am using CentOS 8