Multiple repositories

areyus · December 15, 2023, 1:26pm

Since you are already considering r10k, I think you are on the right track.
I took a look at the installer code, and while I am not 100% sure what those options do, I think they create a local git repo which will be automatically deployed to environments if commits are pushed to it’s branches. From what I understand, this is not what you are looking for.
With r10k, you have one branch in the control repo per environment, and each of them can have it’s own Puppetfile with the modules for that environment. Automatic deployment could be realized via some pipeline functionality by, for example, by SSH-ing into the Foreman system and running r10k.

goeldi · December 21, 2023, 1:55pm

Thank you for your feedback, it really helped me. I looked at it again with a co-worker and we think that a repository can be set up in /etc/puppetlabs/code/environments/ and submodules (i.e. subrepos) within it from the various control_repos in Gitlab and Azure Devops as described here: Git Tools - Submodules. r10k would not be a must at first, but only a possible later option.

What isn’t clear to me yet: I initially imported a repo and then had all classes visible in Foreman. If implemented with git submodules, a local git pull cron job would ensure synchronization of the file system in code/environments/repo/*.

But of course I cannot carry out a manual import into Foreman every time, a class was edited in a git commit. How can I trigger Foreman to reimport the environments after the commits?

The repos/submodules look like this in my head:

└── environments (gitlab main repo)
     ├── team1 (git submodule to gitlab repo of team1)
     ├── team2 (git submodule to azure devops repo of team2)
     ├── team3 (git submodule to azure devops repo of team3)
     ├── team4 (git submodule to azure devops repo of team4)
     └── team5 (git submodule to azure devops repo of team5)

areyus · December 27, 2023, 7:42am

For how to trigger a reimport of the Puppet classes, you can either go with hammer (via the hammer proxy import-classes command) or you can use an API call via the POST /api/smart_proxies/:id/import_puppetclasses or POST /api/smart_proxies/:smart_proxy_id/environments/:id/import_puppetclasses endpoints (See /apidoc on your Foreman server for more infos).
I you realize pulling the repos via cron, I would suggest implementing one of these methods in the same cron job.
You could also make this a part of some CI/CD Pipeline process if you use something like that, but that would require to make the actual git checkout also part of that process.

goeldi · January 9, 2024, 1:53pm

Thank you very much, your description seems to be the solution for me.
But for now I am stuck when I go to Configure → Puppet ENC → Environments → Import → puppet.services.example.com. I get this Error:

ERF12-2749 [ProxyAPI::ProxyException]: Unable to get environments from Puppet ([RestClient::Forbidden]: 403 Forbidden) for proxy https://puppet.services.example.com:8443/puppet

I removed /etc/puppetlabs/code and recreated code from the gitlab repo containing git submodules as follows:

code
├── environments
│   ├── common
│   ├── production
│   ├── webprod
│   └── webtest

I also ran foreman-installer with option --puppet-server-environments-recurse

areyus · January 9, 2024, 2:41pm

That error sounds like some sort of authentication/certificate issue between Foreman and the smart-proxy.
Are there any errors in /var/log/foreman-proxy/proxy.log?

goeldi · January 12, 2024, 8:45am

you were right, I had to edit settings.yml in /etc/foreman-proxy to include all hostnames in the trusted-hosts option (localhost, puppet.services.example.com etc.).

goeldi · January 12, 2024, 8:52am

I installed hammer:

foreman-cli                              3.8.0-1+ubuntu2004
ruby-hammer-cli                          3.8.0-1+ubuntu2004
ruby-hammer-cli-foreman                  3.8.0-1+ubuntu2004
ruby-hammer-cli-foreman-puppet           0.0.6-1
ruby-hammer-cli-foreman-remote-execution 0.2.3-1+ubuntu2004
ruby-hammer-cli-foreman-ssh              0.0.3
ruby-hammer-cli-foreman-tasks            0.0.19-1+ubuntu2004
ruby-hammer-cli-foreman-templates        0.2.0-2
ruby-hammer-cli-foreman-webhooks         0.0.4-1

and I can see the binary with which hammer: /usr/bin/hammer

when I run hammer proxy import-classes I get this error:

Warning: An error occured while loading module hammer_cli_foreman_templates.
Error: No such sub-command ‘proxy’.

This is foreman 3.8.0 on Ubuntu 20.04 and Puppet Opensource 7.27.0

goeldi · January 12, 2024, 9:07am

additionally I have set in /etc/hammer/cli_config.yml the different ssl keys to /etc/puppetlabs/puppet/ssl/certs etc. but hammer tells me these are wrong (although working in foreman ui). I also use letsencrypt for apache proxy. I also tried to use the letsencrypt key to no avail.

areyus · January 12, 2024, 12:01pm

This error is weird. The proxy subcommand should be part part of the “core” hammer-cli-foreman package. Can you paste the full output of hammer --help? I just checked the code to see if the command was recently renamed, but it does not look like it.

This should be unrelated and lead do a different error if it was a problem. Have you set up the letsencrypt cert through foreman-installer? If so, I would assume the installer should also set the settings for hammer.

goeldi · January 12, 2024, 2:12pm

Output of hammer --help:

hammer --help

Warning: Couldn’t load configuration file /etc/hammer/cli_config.yml: (): did not find expected key while parsing a block mapping at line 2 column 1.
NOTE: Gem.gunzip is deprecated; use Gem::Util.gunzip instead. It will be removed on or after 2018-12-01.
Gem.gunzip called from /usr/lib/ruby/vendor_ruby/unicode/display_width/index.rb:5.
Could not load the API description from the server: SSL certificate verification failed
Make sure you configured the correct URL and have the server’s CA certificate installed on your system.