RFC: Running Salt as non-root User

Development RFCs
1

Hi everyone :wave:

With the recent updates to Salt (versions 3006.9 and 3007.1), we’ve taken the opportunity to reassess our current Salt configuration within Foreman. As a result, we’d like to open a discussion regarding the user permissions of the salt-master service and its interaction with Foreman.

From version 3006.3 on, Salt ships their own non-root salt user which runs the salt-master service by default. This leads to permission issues with Foreman at certain points. However, changing it back to running the salt-master as root also leads to issues due to a bug in the Salt Publisher ACL interface which we utilize for our remote execution Salt runner. The bug affects all versions from 3006.3 up to the latest releases 3006.9 and 3007.0/1 (see here and here) and is supposed to be resolved in 3006.10 and 3007.2 (see here).

However, with Salt shipping their own non-root user nowadays, we might think about moving away from runnig Salt as root as well. We propose adding a user foreman-salt to run the salt-master and to interact with it.
We identify the following interfaces which would be affected by such a change:

Salt API

Salt API is used within foreman_salt to read Salt environments and Salt states.

By default, we use PAM as authentication method to access the API. According to the documentation, a new user is added which we use only for Salt API access: saltuser. If the salt-master service shall authenticate the user with PAM, it must run as root (only root can authenticate other users, see link). In a non-root scenario, the salt-master user can only authenticate itself. Therefore, we would have to use the same user to run salt-master and to access the Salt API.

  • Create a user foreman-salt with the following properties:
    • no-login
    • has a password
    • Put the credentials in /etc/foreman-proxy/settings.d/salt.yml
  • Adapt salt-master config:
external_auth:
  pam:
    foreman-salt:
      - '@runner'

Remote Execution Runner (Dispatching Salt Jobs)

We run the salt command as foreman-proxy user to dispatch a remote execution Salt job. In order to execute salt as foreman-proxy, we use the Salt Publisher ACL interface. As mentioned in the introduction, a bug prevents us from calling the salt command if the salt-master runs as root. So, this is for all latest Salt versions broken (probably being fixed in the upcoming Salt releases).

When changing to the non-root foreman-salt user, we could add foreman-proxy to the foreman-salt user group giving it the necessary file permissions to execute the salt command.

  • Add foreman-proxy to the foreman-salt user group

Salt-Call (Dispatching Salt Jobs)

The smart proxy calls the salt command for host provisioning tasks and provides a second option for Salt job execution. It uses sudo to switch to a certain user which is by default root. We change this to foreman-salt:

  • Configure salt_command_user in
    /etc/foreman-proxy/settings.d/salt.yml to be foreman-salt

File Permissions

The foreman-salt user would require certain file permissions:

  • r x - /usr/bin/foreman-node
  • rw - /var/lib/foreman-proxy/salt/ ← read Salt reactors and write the lock file for foreman-node
  • r - /var/share/foreman-proxy/salt ← read autosign grains for host/minion authentication
  • r - /srv/salt ← read Salt states/enviroments and custom Foreman runners
  • r - /etc/pki/katllo/puppet/* ← configured in foreman.yaml for Foreman API access
  • r - /etc/salt/foreman.yaml
  • r - /etc/salt/master.d/foreman.conf
  • rw - /var/cache/salt ← must be configured once after restarting the salt-master first time as foreman-salt.
    Alternatively: delete the dir before restarting the service

foreman-node script uses a lock file which must be accessible by the non-root user too:
→ Change lock file from /var/lock/salt-report-upload.lock to
/var/lib/foreman-proxy/salt/salt-report-upload.lock

I would like to add all the necessary configuration steps for such user in foreman-installer. We could add an option like salt-non-root to decide which steps must be performed. I would propose the option to be true by default, since Salt ships with a non-root user too.
I’ve also heard from the community (see @Jeff_Sparrow in Unique Hosts in Foreman (salt) - #22 by Jeff_Sparrow) that they might keep running salt-master as root. So, this should always be an option still.

I’d love to hear your thoughts! While writing this up, I was also wondering whether we actually need all three interfaces (Salt API, Salt-Call, and REX Runner) or whether we could reduce it to Salt-Call.

2

As I have no real experience with Salt just two ideas/hints.

Did you have a look into keeping the default user and use group permissions or posix ACL to grant file access where needed? I remember back in time there were adjustments for dhcp integration which always caused trouble until it was nicely fixed with ACL.

Keep also SELinux in mind if you change any paths.

1 Like
3

Thanks for your feedback!
I didn’t consider ACLs by now. But might be a nice solution, especially for granting access to key files to perform API requests. Do you know whether its already common in foreman-installer?

4

Thanks for this post. We are in the process of upgrading some 50,000+ minions to 3006.9 (eek), but by the time we are done testing, we’ll probably be on 3006.10. That being said, there are a lot of good considerations here. I know first hand that dealing with the different users, permissions, was a real pain point when setting up salt and foreman. One thing not discussed in here were SSL certs and how the salt user reads them different from the puppet (foreman-proxy) user (not to mention adding in cherrypy SSL too). That was a bit of a pain for us as well. Ultimately, once we moved everything over to salt as root, things became much easier (mostly due to the PAM auth issues you point out). However, all of this changes with one-dir, so making this post/RFC to get ahead of the ball, is great. At this moment, I dont have a lot of input in regards to the direction it should go, just where we’ve been and what worked/didnt work. However, I do agree with your last question. No, I do not think Salt API, Salt-Call, and REX Runner are all necessary. I think just salt-call would suffice. In the near future we will actually be refactoring how it all currently works. Correct me if I am wrong, but I dont think the API is required to get states/environments, so those can be gathered with salt-call as well. And REX… we personally dont really see the need. If we are going to use REX, then we’ll just use ssh directly and not go through salt (plus salt has a cmd.run anyhow)

That being said, I do have some further considerations for any future releases:

  • The reports upload python script needs to using a queuing system for the lock file (we have some 6000 minions per salt master, and were hitting lock issues in about 1/5 hosts). I changed the script to use pythons fcntl locking system, to which (as I understand it and as it appears so far) uses an OS based queueing system automagically. So now, if the lock file is being held, the next minion report goes into a queue until its released. I also added significant amounts of logging to the salt master log, as well as changing it so it handles not just baseline, but state.apply’s as well (for one off state runs). I also changed the api end point to handle report imports from state.apply. Lastly, I made changes to the reactor to simplify its checks, and to handle state.apply’s as well.
  • We had to make some changes to the querying of states as well. Since we use rootfs and gitfs, the current querying system would attempt to go through ALL states in BOTH filesystems. It would eventually time out due to the number of states (and branches) in gitfs. It would be nice to have a way to put those filters into a config file, instead of editing the source code, but I understand that would require quite a bit of work in order to handle the different ways consumers might use salt.
  • I’ve added 4 new additional job templates to handle direct calling of: cmd.run, module.run, runner.run, state.apply and state.highstate.
  • I have also refactored the foreman-node ruby script. This is a pretty basic script, without any sort of queuing system. It is attempting not only to handle ENC queries for each minion, but also grains/facts uploading. Thats too much for it to handle. Thus, we are splitting parts of it out. I have removed any part of the grains/facts uploading that the script was doing, and move those over to a new reactor that checks for salt/minion/*/refresh_grains in the salt master event queue, then kicks off the reactors python script. The new python script now handles all the grains/facts uploading also with a queue based fcntl system.
  • The next thing Ill be working on is trying to better understand the api endpoints for how hosts are created in foreman, with facts/reports uploads. I need to be able to set a organization, location and hostgroup based on certain grains/facts, during creation. In addition, and I have not looked yet at all, but I need to extend support (if it doesnt exist already) to pull states and environments through the api.

I realize some of these are off topic/scope from this conversation, just adding what I have done with the amazing foreman_salt product that already exists! Thank you for your work!

1 Like
5

This is the example I was talking about: puppet-foreman_proxy/manifests/proxydhcp.pp at master · theforeman/puppet-foreman_proxy · GitHub