Multiple repositories

goeldi · December 14, 2023, 10:24am

Problem:

Question: Can I configure foreman/puppet to use multiple git repos?

I need foreman with these options:

puppetdb
puppetserver
puppet manifests / modules managed by git repos
multiple git repos for environments (gitlab, azure devops)

There is only 1 option available for each --puppet-server-git-repo or --puppet-server-git-repo-path

Expected outcome:

5 Teams edit their puppet manifest files in their own Repository on Azure Devops.
1 Team edits the puppet manifest files in its own Gitlab Repository.
Every Team has its own puppet environment in /etc/puppetlabs/code/environments
Foreman is used for reporting and inventory
No provisioning is done via Foreman
Foreman may be but does not have to be the ENC (as in ENC doing only the node classification but not editing the classes/manifests).
Environments in Foreman are updated as soon as a change is made in the environments (I suppose by r10k?).

Foreman and Proxy versions:

Foreman 3.8.0
Foreman Proxy 3.8.0

Foreman and Proxy plugin versions:

ruby-hammer-cli-foreman-puppet 0.0.6-1
ruby-hammer-cli-foreman-webhooks 0.0.4-1
ruby-puppetdb-foreman 6.0.2-1
ruby-hammer-cli-foreman-puppet 0.0.6-1
ruby-foreman-puppet 6.0.1-1

Puppet versions:

Puppet Agent 7.27.0
PuppetDB 7.15.0
PuppetServer 7.14.0

Distribution and version:
Ubuntu 20.04.6

Other relevant data:
/etc/puppetlabs/puppet/auth.conf

path /puppet/v4/environment_classes
method find
allow *

foreman.yaml

---
:url: "https://puppet.services.example.com"
:ssl_ca: "/etc/ssl/certs/ca-certificates.crt"
:ssl_cert: "/etc/puppetlabs/puppet/ssl/certs/puppet.services.example.com.pem"
:ssl_key: "/etc/puppetlabs/puppet/ssl/private_keys/puppet.services.example.com.pem"
:puppetdir: "/opt/puppetlabs/server/data/puppetserver"
:puppetuser: "puppet"
:facts: true
:fact_extension: "json"
:timeout: 60
:report_timeout: 60
:report_retry_limit: 1
:threads: null

puppet.conf

[main]
    basemodulepath = /etc/puppetlabs/code/environments/common:/etc/puppetlabs/code/modules:/opt/puppetlabs/puppet/modules:/usr/share/puppet/modules
    certname = puppet.services.example.com
    codedir = /etc/puppetlabs/code
    environmentpath = /etc/puppetlabs/code/environments
    hiera_config = $confdir/hiera.yaml
    hostprivkey = $privatekeydir/$certname.pem { mode = 640 }
    logdir = /var/log/puppetlabs/puppet
    pluginfactsource = puppet:///pluginfacts
    pluginsource = puppet:///plugins
    privatekeydir = $ssldir/private_keys { group = service }
    reports = foreman,puppetdb
    rundir = /var/run/puppetlabs
    server = puppet.services.example.com
    show_diff = false
    ssldir = /etc/puppetlabs/puppet/ssl
    vardir = /opt/puppetlabs/puppet/cache

[agent]
    classfile = $statedir/classes.txt
    default_schedules = false
    environment = production
    localconfig = $vardir/localconfig
    masterport = 8140
    noop = false
    report = true
    runinterval = 1800
    splay = false
    splaylimit = 1800
    usecacheonfailure = true

[server]
    autosign = /etc/puppetlabs/puppet/autosign.conf { mode = 0664 }
    ca = true
    certname = puppet.services.example.com
    external_nodes = /etc/puppetlabs/puppet/node.rb
    logdir = /var/log/puppetlabs/puppetserver
    node_terminus = exec
    parser = current
    rundir = /var/run/puppetlabs/puppetserver
    ssldir = /etc/puppetlabs/puppet/ssl
    storeconfigs = false
    strict_variables = false
    vardir = /opt/puppetlabs/server/data/puppetserver

puppetdb.conf

[main]
#server_urls = https://puppet.services.example.com:8081
server_urls = https://puppet:8081

routes.yaml

---
master:
  facts:
    terminus: puppetdb
    cache: yaml

areyus · December 15, 2023, 1:26pm

Since you are already considering r10k, I think you are on the right track.
I took a look at the installer code, and while I am not 100% sure what those options do, I think they create a local git repo which will be automatically deployed to environments if commits are pushed to it’s branches. From what I understand, this is not what you are looking for.
With r10k, you have one branch in the control repo per environment, and each of them can have it’s own Puppetfile with the modules for that environment. Automatic deployment could be realized via some pipeline functionality by, for example, by SSH-ing into the Foreman system and running r10k.

goeldi · December 21, 2023, 1:55pm

Thank you for your feedback, it really helped me. I looked at it again with a co-worker and we think that a repository can be set up in /etc/puppetlabs/code/environments/ and submodules (i.e. subrepos) within it from the various control_repos in Gitlab and Azure Devops as described here: Git Tools - Submodules. r10k would not be a must at first, but only a possible later option.

What isn’t clear to me yet: I initially imported a repo and then had all classes visible in Foreman. If implemented with git submodules, a local git pull cron job would ensure synchronization of the file system in code/environments/repo/*.

But of course I cannot carry out a manual import into Foreman every time, a class was edited in a git commit. How can I trigger Foreman to reimport the environments after the commits?

The repos/submodules look like this in my head:

└── environments (gitlab main repo)
     ├── team1 (git submodule to gitlab repo of team1)
     ├── team2 (git submodule to azure devops repo of team2)
     ├── team3 (git submodule to azure devops repo of team3)
     ├── team4 (git submodule to azure devops repo of team4)
     └── team5 (git submodule to azure devops repo of team5)

areyus · December 27, 2023, 7:42am

For how to trigger a reimport of the Puppet classes, you can either go with hammer (via the hammer proxy import-classes command) or you can use an API call via the POST /api/smart_proxies/:id/import_puppetclasses or POST /api/smart_proxies/:smart_proxy_id/environments/:id/import_puppetclasses endpoints (See /apidoc on your Foreman server for more infos).
I you realize pulling the repos via cron, I would suggest implementing one of these methods in the same cron job.
You could also make this a part of some CI/CD Pipeline process if you use something like that, but that would require to make the actual git checkout also part of that process.

goeldi · January 9, 2024, 1:53pm

Thank you very much, your description seems to be the solution for me.
But for now I am stuck when I go to Configure → Puppet ENC → Environments → Import → puppet.services.example.com. I get this Error:

ERF12-2749 [ProxyAPI::ProxyException]: Unable to get environments from Puppet ([RestClient::Forbidden]: 403 Forbidden) for proxy https://puppet.services.example.com:8443/puppet

I removed /etc/puppetlabs/code and recreated code from the gitlab repo containing git submodules as follows:

code
├── environments
│   ├── common
│   ├── production
│   ├── webprod
│   └── webtest

I also ran foreman-installer with option --puppet-server-environments-recurse

areyus · January 9, 2024, 2:41pm

That error sounds like some sort of authentication/certificate issue between Foreman and the smart-proxy.
Are there any errors in /var/log/foreman-proxy/proxy.log?

goeldi · January 12, 2024, 8:45am

you were right, I had to edit settings.yml in /etc/foreman-proxy to include all hostnames in the trusted-hosts option (localhost, puppet.services.example.com etc.).

goeldi · January 12, 2024, 8:52am

I installed hammer:

foreman-cli                              3.8.0-1+ubuntu2004
ruby-hammer-cli                          3.8.0-1+ubuntu2004
ruby-hammer-cli-foreman                  3.8.0-1+ubuntu2004
ruby-hammer-cli-foreman-puppet           0.0.6-1
ruby-hammer-cli-foreman-remote-execution 0.2.3-1+ubuntu2004
ruby-hammer-cli-foreman-ssh              0.0.3
ruby-hammer-cli-foreman-tasks            0.0.19-1+ubuntu2004
ruby-hammer-cli-foreman-templates        0.2.0-2
ruby-hammer-cli-foreman-webhooks         0.0.4-1

and I can see the binary with which hammer: /usr/bin/hammer

when I run hammer proxy import-classes I get this error:

Warning: An error occured while loading module hammer_cli_foreman_templates.
Error: No such sub-command ‘proxy’.

This is foreman 3.8.0 on Ubuntu 20.04 and Puppet Opensource 7.27.0

goeldi · January 12, 2024, 9:07am

additionally I have set in /etc/hammer/cli_config.yml the different ssl keys to /etc/puppetlabs/puppet/ssl/certs etc. but hammer tells me these are wrong (although working in foreman ui). I also use letsencrypt for apache proxy. I also tried to use the letsencrypt key to no avail.

areyus · January 12, 2024, 12:01pm

This error is weird. The proxy subcommand should be part part of the “core” hammer-cli-foreman package. Can you paste the full output of hammer --help? I just checked the code to see if the command was recently renamed, but it does not look like it.

This should be unrelated and lead do a different error if it was a problem. Have you set up the letsencrypt cert through foreman-installer? If so, I would assume the installer should also set the settings for hammer.

goeldi · January 12, 2024, 2:12pm

Output of hammer --help:

hammer --help

Warning: Couldn’t load configuration file /etc/hammer/cli_config.yml: (): did not find expected key while parsing a block mapping at line 2 column 1.
NOTE: Gem.gunzip is deprecated; use Gem::Util.gunzip instead. It will be removed on or after 2018-12-01.
Gem.gunzip called from /usr/lib/ruby/vendor_ruby/unicode/display_width/index.rb:5.
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server’s CA certificate installed on your system.