Setup r10k/g10k and Foreman

Tutorials
1

First of all, thanks to @evgeni for his help about r10k, this guide wouldn’t be here without him

r10k is a tool to deploy Puppet environments and modules, g10k is a fork in go with a focus on performance.

TOC
|- Environment
|- Common steps
__|-- Install r10k
__|-- Dependencies
__|-- Configure sources
__|-- The Hammer
|- One Smart Proxy, on Foreman Host
__|-- Method 1: r10k postrun executes Hammer
__|-- Method 2: r10k postrun executes a custom script
____|— Deploy the custom script
____|— r10k postrun the script
____|— The script
|- Multiple Smart Proxies
__|-- Prerequisites
__|-- Smart Proxies
____|— Edit the Puppet user
__|-- Foreman
____|— Key pair
____|— Deploy the custom script
____|— r10k postrun the script
____|— The script
|- Deploying environments
|- ToDo
|- Feedback
|- Changes

Environment

This tutorial has been tested with EL9/Foreman 3.11.0/Puppet 7/r10k - it may require modifications on different versions/distributions

Common Steps

Install r10k

Install r10k manually with gem install r10k or use the VoxPupuli’s puppet-r10k module

Dependencies

If using Puppet cache, install curl

Configure sources

Edit /etc/puppetlabs/r10k/r10k.yaml (see the official example here) to add at least a valid source:

sources:
  operations:
    remote: 'git@github.com:my-org/org-operations-modules'
    basedir: '/etc/puppetlabs/puppet/environments'

The Hammer

Ensure the Hammer CLI was installed using foreman-installer --foreman-cli-* parameters and has the right credentials in /root/.hammer/cli.modules.d/foreman.yml with --foreman-cli-username, --foreman-cli-password and --foreman-cli-manage-root-config

Puppet in secure_path

Using sudo visudo ensure that Puppet is in secure_path, like:

Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/opt/puppetlabs/bin

One Smart Proxy, on Foreman Host

Method 1: r10k postrun executes Hammer

Simpler to set up, but does not allow for environment cache usage or anything else than a single Hammer CLI action.

Edit /etc/puppetlabs/r10k/r10k.yaml to add a r10k postrun to reload Puppet environments:

postrun: ['hammer', 'proxy', 'import-classes', '--id', '1']
sources:
  operations:
    remote: 'git@github.com:my-org/org-operations-modules'
    basedir: '/etc/puppetlabs/puppet/environments'

Method 2: r10k postrun executes a custom script

Deploy the custom script

Deploy your script manually or through Puppet, here there is an example code with all the necessary information:

file {
  default:
    before => Class['r10k'],
    mode     => '0751',
    group    => 'root',
    owner    => 'root',
    selrange => 's0',
    selrole  => 'object_r',
    seltype  => 'bin_t',
    seluser  => 'system_u',
    ;
  '/opt/foreman-r10k':
    ensure => directory,
    ;
  '/opt/foreman-r10k/foreman-r10k.sh':
    source   => 'edit-your-source-accordingly',
    ;
}

This setup is based on the content of /opt/puppetlabs/puppet

r10k postrun the script

Edit /etc/puppetlabs/r10k/r10k.yaml to add a r10k postrun that runs a custom script:

postrun:
- "/opt/foreman-r10k/foreman-r10k.sh"
sources:
  operations:
    remote: 'git@github.com:my-org/org-operations-modules'
    basedir: '/etc/puppetlabs/puppet/environments'

The script

#!/usr/bin/env bash
# Purge Puppet Server cache
# See https://www.puppet.com/docs/puppet/7/server/admin-api/v1/environment-cache.html
puppet_ca_cert="$(puppet config print cacert)"
puppet_cert="$(puppet config print hostcert)"
puppet_key="$(puppet config print hostprivkey)"
# Proxy name is not guaranteed to be the hostname, take the URL and remove protocol and port
proxy="$(sudo hammer --csv --no-headers proxy list --per-page all --fields url | sed -E 's#https?://##; s#:.*##')"

curl --cert "$puppet_cert" \
    --key "$puppet_key" \
    --cacert "$puppet_ca_cert" \
    -X DELETE "https://$proxy:8140/puppet-admin-api/v1/environment-cache"
# Import classes
hammer proxy import-classes --name "$proxy"

Multiple Smart Proxies

Prerequisites

Install rsync on Foreman and all the Smart Proxies

package { 'rsync':
    ensure => present,
}

And generate a public/private key pair that will be used to use rsync between the hosts

Smart Proxies

Edit the Puppet user

Allow login to the Puppet user and a shell creation. Example with sshkeys_core:

user { 'puppet':
  purge_ssh_keys => true,
  shell          => '/bin/bash',
}

ssh_authorized_key { 'puppet':
  ensure => present,
  user   => 'puppet',
  type   => 'ssh-ed25519',
  key    => 'your-public-key-here',
}

Notes

When testing ssh auth, it may fail because (on EL see /var/log/secure):

  • the puppet user has been locked because it has no password, set it to have no password and allow only SSH login with sudo usermod -p '*' puppet
  • the puppet user home has “wrong” permissions, sudo chmod 750 /opt/puppetlabs/server/data/puppetserver should fix it

Foreman

Key pair

Place the public and private keys in /root/.ssh, and test the ssh connection between the hosts:

ssh -i /root/.ssh/your-private-key -l puppet my-smart-proxy.example.com

This will also allow to accept the smart proxy’s public key.

Deploy the custom script

Deploy your script manually or through Puppet, here there is an example code with all the necessary information:

file {
  default:
    before => Class['r10k'],
    mode     => '0751',
    group    => 'root',
    owner    => 'root',
    selrange => 's0',
    selrole  => 'object_r',
    seltype  => 'bin_t',
    seluser  => 'system_u',
    ;
  '/opt/foreman-r10k':
    ensure => directory,
    ;
  '/opt/foreman-r10k/foreman-r10k.sh':
    source   => 'edit-your-source-accordingly',
    ;
}

This setup is based on the content of /opt/puppetlabs/puppet

r10k postrun the script

Edit /etc/puppetlabs/r10k/r10k.yaml to add a r10k postrun that runs a custom script:

postrun:
- "/opt/foreman-r10k/foreman-r10k.sh"
sources:
  operations:
    remote: 'git@github.com:my-org/org-operations-modules'
    basedir: '/etc/puppetlabs/puppet/environments'

The script

#!/usr/bin/env bash
# List available proxies and remove the Foreman server itself
# It is not safe to make assumptions about id going from 1 to latest_proxy_id,
# as proxies might have been removed/readded, take the URL and remove protocol/port
# as name == hostname is not always true
proxy_list="$(sudo hammer --csv --no-headers proxy list --per-page all --fields url | sed -E 's#https?://##; s#:.*##' | tail -n +2)"
code_dir="$(puppet config print codedir)"

# If there are smart proxies, cycle through them
if [ "$proxy_list" ]; then
    for proxy in $proxy_list
        do
            # Sync environments to the other smart proxies
            if [[ "$proxy" != "$HOSTNAME" ]]; then
                rsync --quiet --recursive --delete \
                --rsh "ssh -i /root/.ssh/your-private-key" \
                "$code_dir/" \
                "puppet@$proxy:$code_dir"
            fi
            # SSH into the proxies and purge cache
            # https://www.puppet.com/docs/puppet/7/server/admin-api/v1/environment-cache.html
            ssh -i /root/.ssh/your-private-key puppet@"$proxy" 'curl \
                --cert /etc/puppetlabs/puppet/ssl/certs/$HOSTNAME.pem \
                --key /etc/puppetlabs/puppet/ssl/private_keys/$HOSTNAME.pem \
                --cacert /etc/puppetlabs/puppetserver/ca/ca_crt.pem \
                -X DELETE "https://$HOSTNAME:8140/puppet-admin-api/v1/environment-cache"'
            # And import the Puppet classes
            hammer proxy import-classes --name "$proxy"
        done
fi

Deploying environments

If everything is setup correctly, sudo r10k deploy environment --modules --generate-types --verbose should deploy all the environments/modules and import them in Foreman (or tell you what’s wrong…)

ToDo

  • Integrate this tutorial into documentation

Feedback

Always welcome, please comment

Changes

2024-07-16_12.30: first version
2024-07-19_10.10: Add single smart proxy script, document test environment, added ToDo
2024-07-19_15.30: add multiple smart proxies instructions, minor improvements to scripts to make them more flexible
2024-07-22_09.40: small script fixes, add cache purge for multiple smart proxies
2024-10-03_16.10: Known issues with SSH auth on smart proxy, edit multiple proxies script to cut first proxy in list (as it’s Foreman itself, it would result in rsync with the same source/destination)

2 Likes
2

Thanks for writing this up. Another idea you can incorporate: puppetserver can cache environments and the installer has --puppet-server-environment-class-cache-enabled true. We don’t enable this by default because you must flush the cache whenever you deploy. That isn’t really controlled by us, but if you set up r10k you can automate it. The API is:

https://www.puppet.com/docs/puppet/latest/server/admin-api/v1/environment-cache.html

For multiple Smart Proxies, you essentially need to change the ID. I didn’t check if it’s supported, but perhaps --name $hostname is also supported and easier to maintain.

It would also be interesting to share some deploy example code. GitHub (and probably other forges) have deployments and it really gives you CI/CD.

More of a development side, but we could authorize the Smart Proxy to use its SSL client certificates to invoke the import-classes API directly. That way we wouldn’t use Hammer with user credentials, but reuse the same credentials that the Smart Proxy already uses to retrieve the ENC and send facts & reports. smart-proxy/extra/puppet_sign.rb at develop · theforeman/smart-proxy · GitHub is an example of reusing the certificates to call to Foreman.

I’ve resisted the urge to build r10k deployment support into the Smart Proxy because it may not share the filesystem with the Puppetserver, but making it easy to flush the environment cache and import classes on Foreman could be done. Even if it’s just some helper script you can easily call.

Last thought: Configuring hosts using Puppet has some mention of r10k, but is rather incomplete.

2 Likes
3

Thanks, I’ll have a look at it. Maybe for one node the easier solution is just to purge the cache directory

I already have a (horrible) script that iterates over the proxies:

! /bin/bash

# Count lines of hammer proxy list
proxyListLines="$(hammer proxy list --per-page all | wc -l)"
# Remove number of lines used as headers etc
proxyNumber=$((proxyListLines - 4))

# If there are smart proxies, cycle through them
if [[ $proxyNumber -ge 1 ]]; then
    # Generate sequence from 1 to higher proxy number
    for i in $(seq 1 $proxyNumber)
        do
            hammer proxy import-classes --id "$i"
        done
fi

But as evgeni/Zhenech pointed on Matrix, Hammer also outputs json/csv that’s easier to iterate, and I need to synchronize the Puppet environments between the proxies with a solution like rsync (and I’m working on it)

At the moment I’m working on some internal services, when I’ll have a solution (and time) I’ll try to publish it on GitHub or similar, ATM this is my attempt to make the “results” public

Interesting but actually my Ruby knowledge is quite limited :sweat_smile:

Maybe I’ll try to do a PR to the docs (are them in a repository?)

4

Yes, they are: GitHub - theforeman/foreman-documentation: Documentation for the Foreman Project and its ecosystem

The file about r10k would be guides/common/modules/proc_managing-puppet-modules-with-r10k.adoc

1 Like
5

No, it’s the in-process memory of the puppetserver that needs to be purged. It won’t even look on disk.

Yes, you can already start with hammer proxy list --per-page all --csv --no-headers. Untested, but I’m guessing something like this probably works:

#!/bin/sh
proxies=$(hammer proxy list --per-page all --csv --no-headers | cut -d, -f1)
for proxy in $proxies ; do
  hammer proxy import-classes --id "$proxy"
done
6

Ok, I’m still missing some Puppet internals

I’ll test ASAP, thanks

7

2024-07-19_10.10: Add single smart proxy script, document test environment, added ToDo

8

2024-10-03: Known issues with SSH auth on smart proxy, edit multiple proxies script to cut first proxy in list (as it’s Foreman itself, it would result in rsync with the same source/destination)