Salt smart proxy plugin configuration help needed

OmniByte · July 23, 2018, 11:21am

Problem:
I’m having a problem to configure a smart proxy in 3-server setup. One machine runns Foreman and uses puppet to manage itself and the other 2 machines. The second machine (salt-master) runs Salt and the third (monitoring) is the icinga monitoring host - For which I have already configured the monitoring plugin successfully (see my other post [1]).
I have difficulties mapping the installation instructions to my setup, specifically which keys to use where. In the doc it uses the subjects" foreman" and “client” and I don’t know who they represent in my setup.
Note: I am working on a new concept for our instrastructure management, that’s way I use puppet and salt as I experiment with both tools. In the final setup, we will choose either Puppet or Salt.

Expected outcome:
On salt-master:
The Installation instructions state that in /etc/salt/foreman.yaml the SSL ca, key and certificate should be in /etc/puppetlabs/ssl, however, in the version of puppet that I run those files are in /var/lib/puppet/ssl. Since the files are organized in a similar fashion though, I configured them as below.
/etc/salt/foreman.yaml
:proto: https
:host: salt-master
:port: 443
:ssl_ca: “/var/lib/puppet/ssl/certs/ca.pem”
:ssl_cert: “/var/lib/puppet/ssl/certs/salt-master.pem”
:ssl_key: “/var/lib/puppet/ssl/certs/private_keys/salt-master.pem”

Also, I applied the same keys for the cherrypy server:

rest_cherrypy:
  port: 9191
  host: 0.0.0.0
  ssl_key: /var/lib/puppet/ssl/private_keys/salt-master.pem
  ssl_crt: /var/lib/puppet/certs/salt-master.pem


I, [2018-07-23T11:55:36.167615 ]  INFO -- : Successfully initialized 'salt'
I, [2018-07-23T11:55:36.167730 ]  INFO -- : Successfully initialized 'foreman_proxy'
W, [2018-07-23T11:55:36.169837 ]  WARN -- : Missing SSL setup, https is disabled.
`I, [2018-07-23T11:58:09.654840 ]  INFO -- : Smart proxy has launched on 1 socket(s), waiting for requests`

On foreman:
In /etc/foreman-proxy/settings.d/salt.yml

:use_api: true
:api_auth: pam
:api_url: https://foreman:9191
:api_username: saltuser
:api_password: saltpassword

Foreman and Proxy versions:

Foreman and Proxy plugin versions:
foreman_proxy: 1.17.1
Other relevant data:
salt-master debug:

 HOSTNAME: salt-master
       OS: debian
  RELEASE: stretch/sid
  FOREMAN:
     RUBY: ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu]
   PUPPET: 3.8.5

foreman debug:

HOSTNAME: foreman
OS: debian
RELEASE: stretch/sid
FOREMAN: 1.17.1
RUBY: ruby 2.3.1p112 (2016-04-26) [x86_64-linux-gnu]
PUPPET: 4.10.12

Thanks for any input in advance.

Nic

[1] https://community.theforeman.org/t/foreman-proxy-plugin-monitoring-with-icinga/10399

aruzicka · July 23, 2018, 12:14pm

Hey,
don’t know much about this but I noticed one thing

Looking at foreman-salt docs, the :host: should be the foreman server. You mentioned foreman and salt-master are different machines, yet when configuring salt you point it towards itself (unless your foreman server is called salt-master).

And the SSL certs configuration looks good to me.

OmniByte · July 23, 2018, 2:48pm

Thanks for the tip, Adam.

I changed the value of the host field but it did’t put things to work. It seems that the problem is that the foreman-proxy is not running, even though WEBrick states so in the logs. If I inspect netstat, I can see that there is no process listening on port 8000 (which I specified to bind on in /etc/foreman-proxy/settings.yml.

The strange thing is, there is no error message in the logs and the service command also reports the service is healthy

root@server1110:/home/support# tail -n20 /var/log/foreman-proxy/proxy.log
I, [2018-07-23T11:58:09.654763 ]  INFO -- : WEBrick::HTTPServer#start: pid=3248 port=8000
I, [2018-07-23T11:58:09.654840 ]  INFO -- : Smart proxy has launched on 1 socket(s), waiting for requests
I, [2018-07-23T13:24:51.198879 ]  INFO -- : going to shutdown ...
I, [2018-07-23T13:24:51.199041 ]  INFO -- : WEBrick::HTTPServer#start done.
I, [2018-07-23T13:24:51.566443 ]  INFO -- : Successfully initialized 'salt'
I, [2018-07-23T13:24:51.566539 ]  INFO -- : Successfully initialized 'foreman_proxy'
W, [2018-07-23T13:24:51.568683 ]  WARN -- : Missing SSL setup, https is disabled.
I, [2018-07-23T13:24:51.568911 ]  INFO -- : WEBrick 1.3.1
I, [2018-07-23T13:24:51.568953 ]  INFO -- : ruby 2.3.1 (2016-04-26) [x86_64-linux-gnu]
I, [2018-07-23T13:24:51.570962 ]  INFO -- : WEBrick::HTTPServer#start: pid=7870 port=8000
I, [2018-07-23T13:24:51.571021 ]  INFO -- : Smart proxy has launched on 1 socket(s), waiting for requests

service:

root@server1110:/home/support# service foreman-proxy status
● foreman-proxy.service - Foreman Proxy
   Loaded: loaded (/lib/systemd/system/foreman-proxy.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2018-07-23 16:30:02 CEST; 6s ago
 Main PID: 32556 (ruby)
   CGroup: /system.slice/foreman-proxy.service
           └─32556 ruby /usr/share/foreman-proxy/bin/smart-proxy --no-daemonize

Jul 23 16:30:00 server1110 systemd[1]: Stopping Foreman Proxy...
Jul 23 16:30:00 server1110 systemd[1]: Stopped Foreman Proxy.
Jul 23 16:30:00 server1110 systemd[1]: Starting Foreman Proxy...
Jul 23 16:30:02 server1110 systemd[1]: Started Foreman Proxy.

Just to make sure it’s not the certificates that mess things up, I disabled https und changed the scheme in all URLs to use “http” instead, but I’m still facing the same issue.

EDIT: output of nmap from a machine in the same network (no firewall)

root@server1116:/home/support# nmap -P0 salt-master

Starting Nmap 7.01 ( https://nmap.org ) at 2018-07-23 16:49 CEST
Nmap scan report for salt-master (10.35.147.110)
Host is up (0.000098s latency).
Not shown: 999 closed ports
PORT   STATE SERVICE
22/tcp open  ssh
MAC Address: 00:50:56:8F:73:BC (VMware)

aruzicka · July 23, 2018, 3:09pm

Just to make sure we’re all on the same page, server1110 is the salt-master and runs the smart-proxy as well, correct?

Could you please post the contents of /etc/foreman-proxy/settings.yml? It seems something isn’t configured right and without seeing the current config it’s hard to tell what’s misconfigured.

OmniByte · July 24, 2018, 9:02am

Hi Adam

Yes, that’s correct, server1110 is the salt-master that also runs smart-proxy. I renamed server11110 to “salt-master” in previous excerpts to give it more semantic but forgot to do it in the last to code blocks. Apologies for the confusion.

Below is the content of the settings.yml from the salt-master:

:settings_directory: "/etc/foreman-proxy/settings.d"
:trusted_hosts: []
:daemon: true
:bind_host:
  - salt-master
:http_port: 8000

I found the problem that caused the smart-proxy not to bind on the IP - the “bind_host” attribute wasn’t properly indented!

Now I can see a process listening on port 8000 and I can add the smartproxy in foreman. Unfortunately, there is still an issue with the configuration, because Cherrypy server isn’t running. I can see that the API isn’t available in the logs:

E, [2018-07-24T10:35:45.060651 84741f7d] ERROR -- : Failed to list environments: Failed to open TCP connection to salt-master:9191 (Connection refused - connect(2) for "salt-master" port 9191)

And the error message in ForemanUI:

But the cherrypy config seems legitimate to me (except for the non-ssl part ):

external_auth:
  pam:
    saltuser:
      - '@runner'

rest_cherrypy:
  port: 9191
  host: 0.0.0.0
  debug: True
  disable_ssl: true

The salt version matches the salt-api version:

support@salt-master:~$ salt-api --version
salt-api 2015.8.8
support@salt-master:~$ salt --version
salt 2015.8.8 (Beryllium)

aruzicka · July 24, 2018, 9:27am

That’s why I don’t like significant whitespace

If you run netstat, is there a process listening on port 9191? If not, it should be brought up by starting salt-api service. The config looks good to me so it should be just a matter of starting the service.

OmniByte · July 24, 2018, 12:52pm

I agree with you, even more if there are no warning messages if there are missing indents.

Indeed that was the missing puzzle piece, I was able to get it running by restarting salt-api.Thanks for all your help man!
At this point I was able to add the smart proxy in my Foreman instance, nice!

May I continue to pick your brain regarding the SSL configuration? After changing the configs () again to use SSL certificates, I still get this “missing SSL” error message in salt smart-proxy log

W, [2018-07-24T14:28:25.705179 ]  WARN -- : Missing SSL setup, https is disabled.

foreman.yaml

:proto: https
:host: foreman
:port: 443
# if using http with ssl certificates
:ssl_ca: "/var/lib/puppet/ssl/certs/ca.pem"
:ssl_cert: "/var/lib/puppet/ssl/certs/salt-master.pem"
:ssl_key: "/var/lib/puppet/ssl/private_keys/salt-master.pem"

master

external_auth:
  pam:
    saltuser:
      - '@runner'

rest_cherrypy:
  port: 9191
  host: 0.0.0.0
  debug: True
  #disable_ssl: true
  # disabled to test non http NO 24 07
  ssl_key: /var/lib/puppet/ssl/private_keys/salt-master.pem
  ssl_crt: /var/lib/puppet/ssl/certs/salt-master.pem

salt.yaml

:enabled: true
:autosign_file: /etc/salt/autosign.conf
:salt_command_user: root
# Some features require using the Salt API - such as listing
# environments and retrieving state info
:use_api: true
:api_url: https://salt-master:9191
:api_auth: pam
:api_username: saltuser
:api_password: saltpassword

I’m not entirely sure this file needs the api user and password, as it was configured to use certificates in rest_cherrypie - right?

Cheers Nic

aruzicka · July 24, 2018, 1:07pm

Sure thing

You get this error when the SSL certificates are not configured for the smart-proxy. You should verify that you properly configured ssl_ca_file, ssl_certificate and ssl_private_key in /etc/foreman-proxy/settings.yml. The values should be the same as in foreman.yaml, just the keys are named somewhat differently.

OmniByte · July 24, 2018, 2:09pm

Thank you so much, Adam it finally worked!

After I added the settings you mentioned to the /etc/foreman-proxy/settings.yml file, the error message disappeared and I could see the content of the certificate in the logs.

I, [2018-07-24T15:43:58.600777 ]  INFO -- : ruby 2.3.1 (2016-04-26) [x86_64-linux-gnu]
I, [2018-07-24T15:43:58.601474 ]  INFO -- :
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 6 (0x6)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=Puppet CA: foreman
        Validity
            Not Before: Jul 22 09:41:33 2018 GMT
            Not After : Jul 22 09:41:33 2023 GMT
        Subject: CN=salt-master
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (4096 bit)

added directives:

:ssl_ca_file: "/var/lib/puppet/ssl/certs/ca.pem"
:ssl_certificate: "/var/lib/puppet/ssl/certs/salt-master.pem"
:ssl_private_key: "/var/lib/puppet/ssl/private_keys/salt-master.pem"

I would have never gotten there without you, as the only indication that the documentation makes regarding this is:

If your Smart Proxy uses SSL, then the certs and key configured in the YAML should be the same ones it uses to talk to Foreman.

One thing I noticed is that the even though the smart proxy is configured to run on port 8000, it only listens on 8443. I’d think that would cause issues when someone already has port 8443 in use.

By the way: I was even able now to import existing minions to Foreman by following the instructions on this (3 years old!) google group post: https://groups.google.com/forum/#!topic/foreman-users/-QQTl5h4zaE

So thanks again for your great support, is there any way I can endorse or give props otherwise?

Nic

aruzicka · July 25, 2018, 7:09am

You’re welcome. There’s no need, I’m just glad you got it working Although if I might request one thing, I’d like you to let us know how your puppet vs salt evaluation went, if you wouldn’t mind