Puppetrun with puppetssh not working

Support
1

Hello,

*I'm having issues with puppetrun and puppetssh *(:puppet_provider:
puppetssh)

Attached foreman debug file.

Env:

OS: redhat
RELEASE: CentOS release 6.6 (Final)
FOREMAN: 1.7.1
RUBY: ruby 1.8.7 (2013-06-27 patchlevel 374) [x86_64-linux]
PUPPET: 3.7.3

Selinux is disabled on all machines and on foreman.

Configured: /etc/foreman-proxy/settings.d/puppet.yml to use puppetssh

I can run remote commands from foreman to the client without any issues
(passwordless ssh Key authentications works) with user root.

When i run Puppetrun from foreman UI it shows that it run successfully:
And i can see that at the /var/log/foreman-proxy/proxy.log file:

192.168.249.91 - - [20/Jan/2015 11:55:02] "POST /run HTTP/1.1" 200 - 0.0038

however it's not running… it fails… and puppet agent is not running on
the client.
Here is the /var/log/messages on the client:

Jan 20 11:02:30 stg-web2 tag_audit_log: type=CRYPTO_KEY_USER
msg=audit(1421751743.419:25456): user pid=30298 uid=0 auid=4294967295
ses=4294967295 msg='op=destroy kind=server
fp=b3:06:c8:b1:ec:3e:c6:68:a7:41:fb:3c:97:c4:75:1d direction=? spid=30298
suid=0 exe="/usr/sbin/sshd" hostname=? addr=192.168.249.91 terminal=?
res=success'
Jan 20 11:02:30 stg-web2 tag_audit_log: type=CRYPTO_KEY_USER
msg=audit(1421751743.419:25457): user pid=30298 uid=0 auid=4294967295
ses=4294967295 msg='op=destroy kind=server
fp=e0:bf:2d:e5:79:f5:3e:f8:c2:ab:2c:3d:09:33:91:d3 direction=? spid=30298
suid=0 exe="/usr/sbin/sshd" hostname=? addr=192.168.249.91 terminal=?
res=success'
Jan 20 11:02:30 stg-web2 tag_audit_log: type=CRYPTO_SESSION
msg=audit(1421751743.421:25458): user pid=30297 uid=0 auid=4294967295
ses=4294967295 msg='op=start direction=from-client cipher=aes128-ctr
ksize=128 spid=30298 suid=74 rport=34565 laddr=192.168.249.178 lport=22
exe="/usr/sbin/sshd" hostname=? addr=192.168.249.91 terminal=? res=success'
Jan 20 11:02:30 stg-web2 tag_audit_log: type=CRYPTO_SESSION
msg=audit(1421751743.422:25459): user pid=30297 uid=0 auid=4294967295
ses=4294967295 msg='op=start direction=from-server cipher=aes128-ctr
ksize=128 spid=30298 suid=74 rport=34565 laddr=192.168.249.178 lport=22
exe="/usr/sbin/sshd" hostname=? addr=192.168.249.91 terminal=? res=success'
Jan 20 11:02:30 stg-web2 tag_audit_log: type=CRYPTO_KEY_USER
msg=audit(1421751743.466:25460): user pid=30297 uid=0 auid=4294967295
ses=4294967295 msg='op=destroy kind=session fp=? direction=both spid=30298
suid=74 rport=34565 laddr=192.168.249.178 lport=22 exe="/usr/sbin/sshd"
hostname=? addr=192.168.249.91 terminal=? res=success'
Jan 20 11:02:30 stg-web2 tag_audit_log: type=CRYPTO_KEY_USER
msg=audit(1421751743.466:25461): user pid=30297 uid=0 auid=4294967295
ses=4294967295 msg='op=destroy kind=server
fp=b3:06:c8:b1:ec:3e:c6:68:a7:41:fb:3c:97:c4:75:1d direction=? spid=30297
suid=0 exe="/usr/sbin/sshd" hostname=? addr=192.168.249.91 terminal=?
res=success'
Jan 20 11:02:30 stg-web2 tag_audit_log: type=CRYPTO_KEY_USER
msg=audit(1421751743.466:25462): user pid=30297 uid=0 auid=4294967295
ses=4294967295 msg='op=destroy kind=server
fp=e0:bf:2d:e5:79:f5:3e:f8:c2:ab:2c:3d:09:33:91:d3 direction=? spid=30297
suid=0 exe="/usr/sbin/sshd" hostname=? addr=192.168.249.91 terminal=?
res=success'
Jan 20 11:02:30 stg-web2 tag_audit_log: type=USER_LOGIN
msg=audit(1421751743.466:25463): user pid=30297 uid=0 auid=4294967295
ses=4294967295 msg='op=login acct=28756E6B6E6F776E207573657229
exe="/usr/sbin/sshd" hostname=? addr=192.168.249.91 terminal=ssh res=
failed'

Here is the /etc/foreman-proxy/settings.d/puppet.yml:

··· --- # Puppet management :enabled: true :puppet_conf: /etc/puppet/puppet.conf # valid providers: # puppetrun (for puppetrun/kick, deprecated in Puppet 3) # mcollective (uses mco puppet) # puppetssh (run puppet over ssh) # salt (uses salt puppet.run) # customrun (calls a custom command with args) :puppet_provider: puppetssh

customrun command details

Set :customrun_cmd to the full path of the script you want to run,

instead of /bin/false
#:customrun_cmd: /bin/false

Set :customrun_args to any args you want to pass to your custom script.

The hostname of the

system to run against will be appended after the custom commands.

#:customrun_args: -ay -f -s

whether to use sudo before the ssh command

:puppetssh_sudo: false

the command which will be sent to the host

:puppetssh_command: /usr/bin/puppet agent --onetime --no-usecacheonfailure

wait for the command to finish (and capture exit code), or detach process

and return 0

Note: enabling this option causes the Foreman web UI to be blocked when

executing puppetrun,

with timeout from the Browser and/or Foreman’s REST client after 60

seconds.
:puppetssh_wait: false

With which user should the proxy connect

:puppetssh_user: root
:puppetssh_keyfile: /etc/foreman-proxy/id_rsa

Which user to invoke sudo as to run puppet commands

#:puppet_user: root

URL of the puppet master itself for API requests

:puppet_url: https://foreman.hosts-app.com:8140

SSL certificates used to access the puppet master API

:puppet_ssl_ca: /var/lib/puppet/ssl/certs/ca.pem
:puppet_ssl_cert: /var/lib/puppet/ssl/certs/foreman.hosts-app.com.pem
:puppet_ssl_key: /var/lib/puppet/ssl/private_keys/foreman.hosts-app.com.pem

Override use of Puppet’s API to list environments, by default it will use

only if

environmentpath is given in puppet.conf, else will look for environments

in puppet.conf

#:puppet_use_environment_api: true

What as i missing here ?
as mentioned above, If SELINUX is disabled

[root@foreman ~]# sestatus
SELinux status: disabled

And on the clients:

[root@stg-web2 ~]# sestatus
SELinux status: disabled
[root@stg-web2 ~]#

Why Would “ssh terminal” fails with root user ?

2

Hey,

> I can run remote commands from foreman to the client without any issues
> (passwordless ssh Key authentications works) with user root.

the problem here is that foreman-proxy does not run under root, but
under foreman-proxy user. You need to setup ssh client for this user on
the proxy.

··· -- Later, Lukas #lzap Zapletal
3

but loe log shows uid=0
and the configuration says : :puppetssh_user: root
anyway does that mean i need to create foreman-proxy user on each node ?

··· On Tue, Jan 20, 2015 at 3:46 PM, Lukas Zapletal wrote:

Hey,

I can run remote commands from foreman to the client without any issues
(passwordless ssh Key authentications works) with user root.

the problem here is that foreman-proxy does not run under root, but
under foreman-proxy user. You need to setup ssh client for this user on
the proxy.

–
Later,
Lukas #lzap Zapletal

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/Hc9jpn0ljEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

4

> but loe log shows uid=0

What log?

> and the configuration says : :puppetssh_user: root

Yes, this is the target system user. The proxy still runs under
foreman-proxy account.

> anyway does that mean i need to create foreman-proxy user on each node ?

No, its on your Foreman Proxy server. The server that is connecting to
your hosts via ssh.

··· -- Later, Lukas #lzap Zapletal
5

As i wrote above the log is /var/log/messages of the agent

Jan 20 11:02:30 stg-web2 tag_audit_log: type=USER_LOGIN
msg=audit(1421751743.466:25463): user pid=30297 uid=0 auid=4294967295
ses=4294967295 msg='op=login acct=28756E6B6E6F776E207573657229
exe="/usr/sbin/sshd" hostname=? addr=192.168.249.91 terminal=ssh
res=failed'
Do you mean that the SSH is done with foreman-proxy user ? (it's not as ar
as i can see)
Please note the logs from the foreman-proxy server (
/var/log/foreman-proxy/proxy.log) and the logs above are form the client…
what is the reason for the failure here (of the SSH using root)

Thank you for your help

··· On Thursday, January 22, 2015 at 4:57:02 PM UTC+2, Lukas Zapletal wrote: > > > but loe log shows uid=0 > > What log? > > > and the configuration says : :puppetssh_user: root > > Yes, this is the *target* system user. The proxy still runs under > foreman-proxy account. > > > anyway does that mean i need to create foreman-proxy user on each node ? > > No, its on your Foreman Proxy server. The server that is connecting to > your hosts via ssh. > > -- > Later, > Lukas #lzap Zapletal >
6

> Thank you for your help

I've already explained this to you. Let me rephrase once again:

Make sure your foreman-proxy user (on your server where your Foreman
Proxy - or Smart Proxy if you want - runs) can connect to root@yournode.

··· -- Later, Lukas #lzap Zapletal
7

why don't you read what i wrote ?

" I can run remote commands from foreman to the client without any issues
(passwordless ssh Key authentications works) with user root."

··· On Thu, Jan 22, 2015 at 5:09 PM, Lukas Zapletal wrote:

Thank you for your help

I’ve already explained this to you. Let me rephrase once again:

Make sure your foreman-proxy user (on your server where your Foreman
Proxy - or Smart Proxy if you want - runs) can connect to root@yournode.

–
Later,
Lukas #lzap Zapletal

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/Hc9jpn0ljEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

8

> " I can run remote commands from foreman to the client without any issues
> (passwordless ssh Key authentications works) with user root."

Again, Foreman does NOT connect to the client when doing puppetssh.
It's the proxy.

Details really matter.

··· -- Later, Lukas #lzap Zapletal
9

Forman and the proxy are the sane server

··· On Fri, Jan 23, 2015, 16:45 Lukas Zapletal wrote:

" I can run remote commands from foreman to the client without any issues
(passwordless ssh Key authentications works) with user root."

Again, Foreman does NOT connect to the client when doing puppetssh.
It’s the proxy.

Details really matter.

–
Later,
Lukas #lzap Zapletal

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/foreman-users/Hc9jpn0ljEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

10

> Forman and the proxy are the sane server

That does not change the fact the two processes are running as two
different effective users.

··· -- Later, Lukas #lzap Zapletal
11

sorry, you are not clear …
foreman-proxy is not relevant to SSH:

foreman-proxy:x:498:498:Foreman Proxy
account:/usr/share/foreman-proxy:/bin/false
foreman:x:497:497:Foreman:/usr/share/foreman:/bin/false

root user can connect from the forman-proxy server to the client without
any problem

can you please explain what is the issue ?

··· On Wed, Jan 28, 2015 at 10:47 AM, Lukas Zapletal wrote:

Forman and the proxy are the sane server

That does not change the fact the two processes are running as two
different effective users.

–
Later,
Lukas #lzap Zapletal

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/Hc9jpn0ljEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

12

> can you please explain what is the issue ?

From the server where you have your foreman-proxy running:

$ su foreman-proxy -s /bin/bash
bash$ ssh root@your_puppet_client "puppet agent --onetime"

If you get this working, the feature you want will work as well.

··· -- Later, Lukas #lzap Zapletal
13

ok but why should it work if foreman-proxy and foreman are installed with
these settings automatically ?
it can't work like this…

cat /etc/passwd
foreman-proxy:x:498:498:Foreman Proxy
account:/usr/share/foreman-proxy:/bin/false
foreman:x:497:497:Foreman:/usr/share/foreman:/bin/false

··· On Wed, Jan 28, 2015 at 6:21 PM, Lukas Zapletal wrote:

can you please explain what is the issue ?

From the server where you have your foreman-proxy running:

$ su foreman-proxy -s /bin/bash
bash$ ssh root@your_puppet_client “puppet agent --onetime”

If you get this working, the feature you want will work as well.

–
Later,
Lukas #lzap Zapletal

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/Hc9jpn0ljEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

14

> it can't work like this…

It can, trust me.

> cat /etc/passwd
> foreman-proxy:x:498:498:Foreman Proxy
> account:/usr/share/foreman-proxy:/bin/false
> foreman:x:497:497:Foreman:/usr/share/foreman:/bin/false

Foreman Proxy drops privileges to foreman-proxy after start. This is the
standard practice. Shell is disabled of course. This is the standard
practice. It's no magic here, really.

Have you ever tried what I tell you to do?

··· > $ su foreman-proxy -s /bin/bash > bash$ ssh root@your_puppet_client "puppet agent --onetime"

–
Later,
Lukas #lzap Zapletal

15

I will try and update
Thanks

··· On Wed, Jan 28, 2015 at 7:09 PM, Lukas Zapletal wrote:

it can’t work like this…

It can, trust me.

cat /etc/passwd
foreman-proxy:x:498:498:Foreman Proxy
account:/usr/share/foreman-proxy:/bin/false
foreman:x:497:497:Foreman:/usr/share/foreman:/bin/false

Foreman Proxy drops privileges to foreman-proxy after start. This is the
standard practice. Shell is disabled of course. This is the standard
practice. It’s no magic here, really.

Have you ever tried what I tell you to do?

$ su foreman-proxy -s /bin/bash
bash$ ssh root@your_puppet_client “puppet agent --onetime”

–
Later,
Lukas #lzap Zapletal

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/Hc9jpn0ljEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

16

please ready the commands below carefully :

simple ssh with key:
[root@foreman foreman-proxy]# ssh -i /etc/foreman-proxy/id_rsa node-name -
Works ok

but the command you suggested does not work, unless i don't use ssh key and
manually enter the root password - what we don't expect foreman-proxy to do
correct ?

ssh -i /etc/foreman-proxy/id_rsa root@node-name "puppet agent --onetime" -
This does not work and fails.

This is why :puppet_provider: puppetssh fails

··· On Wed, Jan 28, 2015 at 7:39 PM, haim ari wrote:

I will try and update
Thanks

On Wed, Jan 28, 2015 at 7:09 PM, Lukas Zapletal lzap@redhat.com wrote:

it can’t work like this…

It can, trust me.

cat /etc/passwd
foreman-proxy:x:498:498:Foreman Proxy
account:/usr/share/foreman-proxy:/bin/false
foreman:x:497:497:Foreman:/usr/share/foreman:/bin/false

Foreman Proxy drops privileges to foreman-proxy after start. This is the
standard practice. Shell is disabled of course. This is the standard
practice. It’s no magic here, really.

Have you ever tried what I tell you to do?

$ su foreman-proxy -s /bin/bash
bash$ ssh root@your_puppet_client “puppet agent --onetime”

–
Later,
Lukas #lzap Zapletal

–
You received this message because you are subscribed to a topic in the
Google Groups “Foreman users” group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/foreman-users/Hc9jpn0ljEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

17

> ssh -i /etc/foreman-proxy/id_rsa root@node-name "puppet agent --onetime" -
> This does not work and fails.

This is what our proxy does:

$ su foreman-proxy -s /bin/bash
bash$ ssh root@your_puppet_client "puppet agent --onetime"

Make sure it is working by creating /usr/share/foreman-proxy/.ssh
directory with proper permissions, SELinux label and keys/athorized_keys
in it. Test it first with the command(s) above. Then it will work.

··· -- Later, Lukas #lzap Zapletal
18

After reading through this discussion, I managed to get puppetssh working
inside my packer/vagrant environment. You can view the code here:

https://github.com/Sher-Chowdhury/vagrant-foreman/tree/3cfe96ef8a8cf16d60e820274e790d4c0e686286

Note: the above link is for a particular commit, rather than the latest
version of the code. That's because I'm going to switch from puppetssh to
mcollective in the near future.

The key thing that got it working for me is that I needed to reboot my
foreman vm after I enabled puppetssh, otherwise it would carry on trying to
do "puppet kick" rather than puppetssh. another thing that was blocking my
is that when the puppet-run keeps getting blocked by the ssh RSA
fingerprint authentication prompt that happens to the foreman-proxy user
when it tries to auto-connect to a puppet agent for the first time. In my
case I created the ~/.ssh/config file and specified it to auto-accept all
RSA authentication prompts. This allowed the puppetssh run to occur without
any interuptions.