Problem configuring Salt proxy on remote Salt master

dhs-rec · March 4, 2021, 9:38am

Problem:
I’ve got a running Foreman/Puppet installation to manage our environment. Foreman is mainly used a means to visualize Puppet data (reports, facts). It runs on a single machine (“puppet .domain.com”). Some time ago we’ve also set up a Salt master server on a different machine (“salt .domain.com”). It also has the Puppet agent installed, which was used to configure the Salt master.

Now I wanted to add this Salt master to the Foreman setup running on the Puppet host (both are Debian Buster). So I first installed the ruby-foreman-salt package on the Puppet host, followed by installation of ruby-smart-proxy-salt and salt-api on the Salt host.

On the latter, I also changed the following files:

/etc/salt/foreman.yaml:

:proto: https
:host: puppet .domain.com
:port: 443
:ssl_ca: “/etc/puppetlabs/puppet/ssl/certs/ca.pem”
:ssl_cert: “/etc/puppetlabs/puppet/ssl/certs/salt.domain.com.pem”
:ssl_key: “/etc/puppetlabs/puppet/ssl/private_keys/salt.domain.com.pem”
:timeout: 10
/usr/bin/salt
:upload_grains: true
/etc/foreman-proxy/settings.yml:

:settings_directory: “/etc/foreman-proxy/settings.d”
:trusted_hosts:
- puppet .domain.com
:daemon: true
:bind_host:
- 0.0.0.0
:http_port: 8000
:ssl_ca_file: “/etc/puppetlabs/puppet/ssl/certs/ca.pem”
:ssl_certificate: “/etc/puppetlabs/puppet/ssl/certs/salt.domain.com.pem”
:ssl_private_key: “/etc/puppetlabs/puppet/ssl/private_keys/salt.domain.com.pem”
/etc/foreman-proxy/settings.d/salt.yml:

:enabled: true
:autosign_file: /etc/salt/autosign.conf
:salt_command_user: root
:use_api: true
:api_url: https://localhost:9191
:api_auth: pam
:api_username: saltuser
:api_password: secretpassword
/etc/salt/master.d/api.conf:

external_auth:
pam:
saltuser:
- ‘@runner’
rest_cherrypy:
port: 9191
host: 0.0.0.0
ssl_key: /etc/puppetlabs/puppet/ssl/private_keys/salt.domain.com.pem
ssl_crt: /etc/puppetlabs/puppet/ssl/certs/salt.domain.com.pem
/etc/salt/master.d/local.conf:

ext_pillar:
- puppet: /usr/bin/foreman-node
master_tops:
ext_nodes: /usr/bin/foreman-node
permissive_pki_access: True
…

Note 1: All the SSL keys/certs above are those of the Puppet agent on salt .domain.com, since this is how I understood the documentation.
Note 2: This editor eats indentation, regardless of whether using block quote or preformated text. None works as expected. So consider all files properly indented.

I finally restarted the salt-master and and foreman-proxy services on salt .domain.com as well as foreman and foreman-proxy on puppet .domain.com.

I could then add the Salt proxy in the Foreman UI, but running foreman-node (on salt .domain.com) fails with an SSL error:

# foreman-node minion .domain.com
Couldn’t retrieve ENC data: Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)

and none of the Salt minions show up in the Foreman UI.

Expected outcome:
Minions should show up in the Foreman UI

Foreman and Proxy versions:
Foreman: 2.3.3
Salt Proxy: 13.2.4

Foreman and Proxy plugin versions:

# dpkg --list|grep proxy-salt
ii ruby-smart-proxy-salt 3.1.2-1 all SaltStack Plug-In for Foreman’s Smart Proxy
ii ruby-smart-proxy-salt-core 0.0.3-1 all SaltStack Plug-In core for Foreman’s Smart Proxy

Distribution and version:
Debian 10.8 (Buster)

Other relevant data:
Not sure what to put here in this case. Please ask.

Note 3: I needed to add blanks to all hostnames, as otherwise the were interpreted as links and summed up to more than 5.

Thanks in advance…

bastian-src · March 5, 2021, 7:25am

Did you add your Salt host as Smart Proxy to Foreman? Such that is is listed under Infrastructure > Smart Proxies with the feature Salt?

Moreover, you should be able to run upload-salt-reports on your Salt host - does it throw an error?

dhs-rec · March 5, 2021, 7:48am

Yes, did that.

Doesn’t throw an error, but also doesn’t upload anything.

dhs-rec · March 5, 2021, 8:00am

Correction: After running a highstate and trying again, it prints this error:

# python3 /usr/sbin/upload-salt-reports
Unable to upload job - aborting report upload
b'{\n  "error": {"message":"Access denied","details":"Missing one of the required permissions: create_reports","missing_permissions":["create_reports"]}\n}\n'

And this shows up in the Foreman log:

2021-03-05T09:02:24 [I|app|153353b8] Started POST "/salt/api/v2/jobs/upload" for 127.0.0.1 at 2021-03-05 09:02:24 +0100
2021-03-05T09:02:24 [I|app|153353b8] Processing by ForemanSalt::Api::V2::JobsController#upload as JSON
2021-03-05T09:02:24 [I|app|153353b8]   Parameters: {"job"=>"[FILTERED]", "apiv"=>"v2"}
2021-03-05T09:02:24 [W|app|153353b8] No SSL cert with CN supplied - request from 10.88.2.230
2021-03-05T09:02:24 [I|app|153353b8]   Rendering api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout
2021-03-05T09:02:24 [I|app|153353b8]   Rendered api/v2/errors/access_denied.json.rabl within api/v2/layouts/error_layout (Duration: 7.2ms | Allocations: 5588)
2021-03-05T09:02:24 [I|app|153353b8] Filter chain halted as #<Proc:0x000055ce2d8da238@/usr/share/foreman/app/controllers/concerns/foreman/controller/smart_proxy_auth.rb:14> render
ed or redirected
2021-03-05T09:02:24 [I|app|153353b8] Completed 403 Forbidden in 24ms (Views: 12.7ms | ActiveRecord: 2.8ms | Allocations: 13726)

dhs-rec · March 5, 2021, 8:28am

BTW: The script is also executed with Python 2 ("#!/usr/bin/env python"), and therefor also fails with the following error unless explicitely executed with Python 3, as above.

Traceback (most recent call last):
  File "/usr/sbin/upload-salt-reports", line 22, in <module>
    import salt.config
ImportError: No module named salt.config

I guess this is because Salt is installed from the “py3” packages.

bastian-src · March 5, 2021, 10:57am

The ssl certs used for the request made by upload-salt-reports are defined in /etc/salt/foreman.yaml on your salt master. Is it possible that there are some flaws with those certs/the key?

In case you are on a test system, you could check the rest of your configuration by setting the :proto: option in that file to http and provide :username: and :password: (foreman login credentials) parameters instead of :ssl_cert: and :ssl_key:.

dhs-rec · March 5, 2021, 11:16am

Not that I would know. Puppet communication works fine with them.

Doing so results in a stacktrace again:

# python3 /usr/sbin/upload-salt-reports
Traceback (most recent call last):
  File "/usr/sbin/upload-salt-reports", line 156, in <module>
    upload(jobs_to_upload())
  File "/usr/sbin/upload-salt-reports", line 121, in upload
    config[':password']))
  File "/usr/lib/python3.7/base64.py", line 58, in b64encode
    encoded = binascii.b2a_base64(s, newline=False)
TypeError: a bytes-like object is required, not 'str'

bastian-src · March 9, 2021, 8:30am

I justed tested the script with http myself and added a PR to fix the issue. Thanks for your help on this!
You could also apply the changes of the PR to /usr/sbin/upload-salt-reports manually & test it again, since it takes a new release to be available in the official package.

If the http request works, we can narrow the error down to the https request (which the errors already point to).

dhs-rec · March 9, 2021, 11:25am

This got me a bit further. I now get the following server side error:

2021-03-09T12:16:18 [I|app|a1e329c4] Started POST "/salt/api/v2/jobs/upload" for 127.0.0.1 at 2021-03-09 12:16:18 +0100
2021-03-09T12:16:18 [I|app|a1e329c4] Processing by ForemanSalt::Api::V2::JobsController#upload as JSON
2021-03-09T12:16:18 [I|app|a1e329c4]   Parameters: {"job"=>"[FILTERED]", "apiv"=>"v2"}
2021-03-09T12:16:18 [I|app|a1e329c4] Processing job 20210309110938502571 from Salt.
2021-03-09T12:16:18 [W|app|a1e329c4] Action failed
2021-03-09T12:16:18 [I|app|a1e329c4] Backtrace for 'Action failed' error (RuntimeError): The Dynflow world was not initialized yet. If your plugin uses it, make sure to call Rails.application.dynflow.require! in some initializer
... [Stacktrace]
2021-03-09T12:16:18 [I|app|a1e329c4]   Rendering api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout
2021-03-09T12:16:18 [I|app|a1e329c4]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (Duration: 4.5ms | Allocations: 5563)
2021-03-09T12:16:18 [I|app|a1e329c4] Completed 500 Internal Server Error in 25ms (Views: 9.1ms | ActiveRecord: 2.5ms | Allocations: 23576)

bastian-src · March 9, 2021, 1:56pm

Dynflow is actually a necessary package of foreman_salt and should therefore be already installed on your Foreman. Could you pls check whether it is enabled on your foreman host via foreman-maintain service list?

dhs-rec · March 9, 2021, 2:26pm

No, it’s not. I’ve installed from packages (see above), so I expected all dependencies to be pulled in automatically. However, if I now try to install ruby-foreman-dynflow, it wants to uninstall ruby-foreman-salt again:

# apt-get install ruby-foreman-dynflow
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following package was automatically installed and is no longer required:
   ruby-foreman-deface (1.5.3-1)
Use 'apt autoremove' to remove it.
The following packages will be REMOVED:
   ruby-foreman-remote-execution (4.2.2-1)
   ruby-foreman-salt (13.2.4-1)
   ruby-foreman-tasks (3.0.3-1)
The following NEW packages will be installed:
   ruby-foreman-dynflow (0.8.6-1)
0 upgraded, 1 newly installed, 3 to remove and 4 not upgraded.
Need to get 1,084 kB of archives.
After this operation, 26.6 MB disk space will be freed.
Do you want to continue? [Y/n] n
Abort.

Or do you want me to install the dynflow smart proxy (ruby-smart-proxy-dynflow) instead?

bastian-src · March 9, 2021, 2:37pm

The related process is called smart_proxy_dynflow_core.service so I would go with ruby-smart-proxy-dynflow. But it should not remove those three packages (tasks, remote-execution, salt) - they are all meant to use dynflow actually.

dhs-rec · March 9, 2021, 3:01pm

Installed that and started the service (and also restarted foreman and foreman-proxy), but the error stays the same.

bastian-src · March 10, 2021, 8:13am

Did you restart using foreman-maintain service restart?

Is something written in the dynflow log file: /var/log/foreman-proxy/smart_proxy_dynflow_core.log?

Another approach on checking dynflow would be to check whether the ruby-gem was installed properly. Therefore, on CentOS the command scl enable tfm bash adds the appropriated PATHs such that you can call gem list | grep dynflow to see the installed gems. Idk whether there is a Debian alternative for the PATHs, nevertheless you can check the installation folder (should be /opt/theforeman/tfm/root/usr/share/gems/gems/) for the entries dynflow, smart_proxy_dynflow and smart_proxy_dynflow_core.

dhs-rec · March 10, 2021, 8:47am

No, I don’t have that command. I’ve restarted them using systemctl restart <service> (as I do since years), since I’ve installed everything from packages.

No, it’s empty.

I don’t have /opt/theforeman. The packages install into the expected standard locations, so everything is supposed to be in the PATH automatically. These are the installed packages:

# dpkg --list|grep dynflow
ii  ruby-dynflow                   1.4.6-1                      all          DYNamic workFLOW orchestration engine
ii  ruby-smart-proxy-dynflow       0.3.0-1                      all          Dynflow runtime for Foreman smart proxy
ii  ruby-smart-proxy-dynflow-core  0.3.2-1                      all          Dynflow runtime for Foreman smart proxy

bastian-src · March 10, 2021, 1:27pm

Ok, so I found the line in foreman_salt where the error is thrown. It occurs when foreman-tasks (which relies on dynflow) is called. I don’t have any experience with Foreman on Debian, so are there maybe any other installable ~foreman-tasks packages in your repo (besides the on you already have)?

Can you see messages in the production log with the tag dyn?

dhs-rec · March 10, 2021, 1:44pm

No. ruby-foreman-tasks and ruby-foreman-tasks-core are the only ones and they’re both installed. The packages are all installed from the official Foreman package repository (http://deb.theforeman.org), btw.

No, not a single one.

dhs-rec · March 11, 2021, 11:45am

Short update: It seems to at least create the Salt minion hosts in Foreman and upload their grains, but not the reports.