RFC: Integrating bolt into remote execution

Community
, ,
1

Hi!

I would like to add support for bolt tasks to the remote execution architecture in foreman.

What’s bolt?
An open source tool from same people as Puppet ( Welcome to Puppet Bolt®). It supports runnings commands (shell or powershell), scripts (anything that’s executeable), tasks or plans.

What’s a task?
A task is anything that’s executeable + a json file describing the purpose of a task and optional input parameters with datatypes. Tasks return json. bolt has a CLI interface and an API, no web UI. bolt can use different transport protocols, ssh and winrm are available by default. It has a plugin system.

Rationale:
Puppet Enterprise (costs per node per year) has a Web UI to start tasks. I think it would be a great fit for the foreman UI to also support bolt tasks. bolt tasks are heavily used within the Puppet Enterprise and Puppet Opensource ecosystem for orchestration. For example OS patching whole clusters or doing software rollouts.

I came up with a very professional drawing:

bolt+foreman.drawio2
bolt+foreman.drawio2752×672 57 KB

I don’t understand all the pieces in the remote execution architecture yet, but my rough ideas:

  • There’s already support for remote execution via ssh, triggered from a system running foreman-proxy. I want to build upon this / extend it.
  • Foreman supports different job categories, ‘bolt tasks’ could be another one?
  • foreman asks foreman-smartproxy which talks to a local bolt-server to get a list of tasks + description, as an alternative to the current job templates.
  • After a user selected a task, they can enter the different parameters, data is send to the smartproxy and then to bolt-server
  • bolt reuses the ssh authentication credentials from the ssh backend
  • all targets, where a task was executed, return structured data, bolt-server itself returns a result set to foreman-proxy which in turn hands it over to foreman

Those are some rough ideas that I want to work out in the next time. If you’ve any ideas, feedback, concerns, let me know.

3 Likes
2

having seen your initial concept before you posted it, this would be a fantastic edition to foreman, and would be a big step (if not a nail in the coffin) for the desire for puppet enterprise.

I’ll be following this with great interest, and if there is anything I can do to offer help or support I’ll be happy to do so

I ike the idea of it being a foreman proxy capability pointing at bolt directly similar to other foreman-proxy functions.

One area I was initially unclear on and ‘wrong’ about was where bolt parameters would be stored, I expected them to be stored within foreman similar to smart class parameters, but I believe the initial concept is to keep them within the bolt task in the puppet module itself.

3

Yes, that’s my idea. Bolt has an API endpoint to list all tasks or a specific task with details

[
  {
    "name": "exec"
  },
  {
    "name": "facter_task"
  },
  {
    "name": "facts"
  },
  {
    "name": "node_manager::update_classes"
  },
  {
    "name": "puppet_agent::facts_diff"
  }
]

{
  "metadata": {
    "description": "Run the Puppet agent facts diff action",
    "parameters": {
      "exclude": {
        "description": "Regex used to exclude specific facts from diff",
        "type": "Optional[String]"
      }
    },
    "files": [
      "puppet_agent/files/rb_task_helper.rb"
    ]
  },
  "name": "puppet_agent::facts_diff",
  "files": [
    {
      "filename": "facts_diff.rb",
      "sha256": "aebed81513df6b0f6534bac5c964f03433416a34fbbf0dfd8a823b12563d5c54",
      "size_bytes": 2058,
      "uri": {
        "path": "/puppet/v3/file_content/tasks/puppet_agent/facts_diff.rb",
        "params": {
          "environment": "peadm"
        }
      }
    },
    {
      "filename": "puppet_agent/files/rb_task_helper.rb",
      "sha256": "4209800a8867439c5271535e4dad20e6fe858c16585f607897b3178047df8d65",
      "size_bytes": 1824,
      "uri": {
        "path": "/puppet/v3/file_content/modules/puppet_agent/rb_task_helper.rb",
        "params": {
          "environment": "peadm"
        }
      }
    }
  ]
}
4

As like most open source tools, bolt lacks some details about using the bolt-server / API. But in case people want to play around:

The dockerfile can be found at bolt/Dockerfile.bolt-server at main · puppetlabs/bolt · GitHub

You can use the following to start it (assumes you’re on a puppetserver with deployed code):

get the code:

git clone https://github.com/puppetlabs/bolt/
cd bolt

copy certs into the container

cp /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem spec/fixtures/ssl/ca.pem
cp /etc/puppetlabs/puppet/ssl/private_keys/$(hostname -f).pem spec/fixtures/ssl/key.pem
cp /etc/puppetlabs/puppet/ssl/certs/$(hostname -f).pem spec/fixtures/ssl/cert.pem

update the config to point to a local running puppetserver:

sed --in-place "s/spec_puppetserver_1/$(hostname -f)/g" config/docker.conf

to mimic the same behaviour as the pe-bolt-server, we do a bind mount to the same dir as PE uses:

docker buildx build --file Dockerfile.bolt-server -t bolt-server .
docker run --network=bridge -itd --name=bolt-server --publish-all -v /etc/puppetlabs/code/:/etc/puppetlabs/code bolt-server

and to test it:

List all tasks

curl --cacert /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem \
--cert /etc/puppetlabs/puppet/ssl/certs/$(hostname -f).pem \
--key /etc/puppetlabs/puppet/ssl/private_keys/$(hostname -f).pem \
https://$(hostname -f):32770/tasks?environment=production --silent | jq .

List one task:

curl --cacert /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem --cert /etc/puppetlabs/puppet/ssl/certs/$(hostname -f).pem --key /etc/puppetlabs/puppet/ssl/private_keys/$(hostname -f).pem https://$(hostname -f):32768/tasks/puppet_agent/facts_diff?environment=peadm --silent | jq .

list all plans:

curl --cacert /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem --cert /etc/puppetlabs/puppet/ssl/certs/$(hostname -f).pem --key /etc/puppetlabs/puppet/ssl/private_keys/$(hostname -f).pem https://$(hostname -f):32768/plans?environment=peadm --silent | jq .

list all plans with details:

curl --cacert /etc/puppetlabs/puppet/ssl/ca/ca_crt.pem --cert /etc/puppetlabs/puppet/ssl/certs/$(hostname -f).pem --key /etc/puppetlabs/puppet/ssl/private_keys/$(hostname -f).pem https://$(hostname -f):32768/plans?environment=peadm\&metadata=true --silent | jq .

some docs: bolt/developer-docs/bolt-server/api.md at main · puppetlabs/bolt · GitHub

5

Would this be a part of the “triggering a job” flow or would this be a one-time action, similar to how foreman “imports” ansible roles?

6

I think it should be part of this in my opinion, because the tasks/plans can change on a regular basis. If this is a one-time thing, we need some kind of cache invalidation to update it from time to time. But maybe this also needs some performance testing.

7

What I would be worried about:

  • it might not fit well into how the ui currently works. You should be able to somehow bend it to your will, but there might be dragons
  • the whole behind-the-scenes machinery is tied heavily to job templates, which I’m not exactly sure what role they would play here
  • a single job can reach different hosts through different proxies
  • a proxy to be used for a given host is not known in advance

I’m not trying to deter you from it, just pointing out where I see potential problems so that you can prepare for them up front if you decide to go ahead with this.

8

Thanks for the great feedback,

initially I was thinking about a dedicated foreman/foreman-proxy addon for bolt support, that doesn’t rely/reuse the existing remote execution. I’m still unsure about the pros vs cons here.

9

Oh! Looks like I took a mental shortcut somewhere. Yeah, that could be an option too.