Foreman-installer scenario for foreman-proxy installation

Due to EL 7 deprecation and no really feasible and trustworthy migration options available I am looking into reinstalling our foreman system consisting of one main server and two proxies, one for content and one for puppet.

Looking at current installation docs for the proxies I noticed that there still doesn’t seem to be a foreman-installer scenario to install a foreman-proxy without content, e.g. for puppet.

The “old” docs have 4.3 Smart Proxies which is pretty much a manual installation without foreman-installer.

The “new” docs have Enabling Puppet Integration with Foreman but those are instructions to enable puppet into an existing proxy.

I don’t really want to install a content proxy, enable puppet and then disable all the content functions to get a puppet only proxy. It’s what I did to install my current puppet proxy but it doesn’t really make much sense to me to go this way and I suspect that Smart Proxy page is empty may be caused by that kind of installation.

But there doesn’t seem to be any instructions telling how to install a smart proxy for puppet without content proxy using foreman-installer. Is that right?

I am unsure how well this describes your scenario, but you could have a look at our downstream orcharhino docs on installing smart proxies: orcharhino Proxy Installation Guide - orcharhino documentation

These docs include some downstream product specific assumptions. In particular you should ignore the bits about “Squid Proxy”, as well as about the “orcharhino Proxy Content job template” (neither is available for upstream Foreman). However, assuming your Proxy has access to the same package sources as your server, these are instructions that use the foreman-installer. You should probably also re-use your foreman installer options used on your server for the proxy installation (in addition to any proxy specific options mentioned in the docs).

I am also going to ping @maximilian to keep an eye on this docs thread.

Well, that is what I already know. But I don’t want to install full content proxy only if I don’t need it. I want a smart proxy without content and only with puppet. In my old installation I used the content proxy installation and then disabled all content functions. But that’s kind of weird and it also installs a lot of pulp packages I don’t need on a puppet only smart proxy.

So, what I am looking for is a simple proxy installer scenario where I could then add additional functions as required. Kind of this:

# foreman-installer --scenario foreman-proxy ...

Maybe it’s not really possible anymore as it seems katello starts to require all smart proxies to have pulp running, even if they don’t have any content. That seems to be the problem with Smart Proxy page is empty and maybe also Updating certificates on content proxy.

I thought it was possible to freely pick and choose some complex set of options for the installer (like leaving away all the content feature options, but adding the options for Pulp etc.). However, I don’t know how it is with the latest Foreman releases, and I also don’t know to what extent Katello expects you to “always have pulp on the proxy” (even if you don’t plan on using it), so you may well be right that this isn’t possible right now.

I am sorry @gvde but I am not too involved with the Foreman/Katello and Smart Proxy installation guides because we use a different approach in our downstream product.

O.K. I haven’t tried a new installation, yet, but I think originally the foreman-installer-katello package had a dependency to pulp so it was all or nothing… But I guess I have to try…

So I guess there is nothing at the moment. During updating the certificates for my “puppet proxy” which was a content proxy with the content stuff remove, I had to reenable again because otherwise the installer wouldn’t configure/replace the certificate. See Updating certificates on content proxy

@katello @installer Does anyone know if there is some installation scenario for Katello + smart proxy without Pulp? Do you think there is a use case for it?

Well, the alternative is probably following the docs here

https://theforeman.org/manuals/3.2/index.html#4.3.1SmartProxyInstallation

which are not very nice. foreman-installer can set all that up with options, but you need to use a fitting scenario and there isn’t one…

O.K. Looking into foreman 3.3 I have found those (new?) docs: Installing an External Smart Proxy Server 3.3

It gets me much close to my goal of a smart proxy for a puppet master without all the content proxy stuff.

Those docs seem to need some work, though.

First: I assumed from the text that the system doesn’t have a puppet agent installed before the installation. If you don’t the proxy server doesn’t have a puppet cert and foreman-installer will fail (unless you make the proxy to a puppet ca as well). This to prep you can set it up before:

cat >> /etc/puppetlabs/puppet/puppet.conf <<EOF
[main]
server = foreman8.example.com
EOF
[root@foreman8-puppet ~]# /opt/puppetlabs/bin/puppet ssl bootstrap 
...

And sign on the main/puppet ca server:

[root@foreman8 ~]# puppetserver ca sign --certname foreman8-puppet.example.com

The installer command doesn’t work as shown:

https://docs.theforeman.org/3.3/Installing_Proxy/index-foreman-el.html#installing-an-external-smart-proxy-upstream_smart-proxy

I need to add

  --puppet-ca-server=foreman8.example.com \
  --no-enable-foreman-plugin-puppet \
  --no-enable-foreman-cli-puppet \

to get it working at all.

O.K. Certificates seem to be a bit more tricky, as the certs plugin isn’t there, thus there is no --certs-tar-file to load certificates generated with foreman-proxy-certs-generate on the main server. I also use server certificates from an external ca, thus I had to add:

  --foreman-proxy-ssl-ca=/etc/foreman-proxy/ssl_ca.pem \
  --foreman-proxy-ssl-cert=/etc/foreman-proxy/ssl_cert.pem \
  --foreman-proxy-ssl-key=/etc/foreman-proxy/ssl_key.pem \

using files which I set up manually:

  • /etc/foreman-proxy/ssl_ca.pem is simply a copy of the same file on the main server (foreman’s “internal” CA root)
  • ssl_key.pem is the private key for the certificate issued by the external CA
  • ssl_cert.pem is the certificate from the external CA

This lets me register the proxy with the main server.

Now am I stuck with getting the puppet client connect to the main server:

[root@foreman8-puppet ~]# /opt/puppetlabs/bin/puppet agent --test --noop
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Failed to find foreman8-puppet.example.com via exec: Execution of '/etc/puppetlabs/puppet/node.rb foreman8-puppet.example.com' returned 1: 
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node foreman8-puppet.example.com: Failed to find foreman8-puppet.example.com via exec: Execution of '/etc/puppetlabs/puppet/node.rb foreman8-puppet.example.com' returned 1: 
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
[root@foreman8-puppet ~]# /etc/puppetlabs/puppet/node.rb foreman8-puppet.example.com
During fact upload occured an exception: SSL_read: tlsv1 alert unknown ca
Serving cached ENC: Could not send facts to Foreman: SSL_read: tlsv1 alert unknown ca
Unable to read from Cache file: No such file or directory @ rb_sysopen - /opt/puppetlabs/server/data/puppetserver/yaml/foreman/foreman8-puppet.example.com.yaml

I need to set up the puppet client certs like those on the main server in /etc/pki/katello/puppet/.

O.K. It’s too bad that the foreman scenario doesn’t know how to handle the certs tar from katello. I think that is how it’s supposed to be. To get it working I had to get certificates by hand.

On the main foreman/katello server generate the certs tar:

[root@foreman8 ~]# foreman-proxy-certs-generate \
    --foreman-proxy-fqdn "foreman8-puppet.example.com" \
    --certs-tar  "~/foreman8-puppet.example.com-certs.tar" \
    --server-cert "/root/certs/foreman8-puppet/foreman8-puppet.cer" \
    --server-key "/root/certs/foreman8-puppet/foreman8-puppet.key" \
    --server-ca-cert "/root/certs/foreman8-puppet/foreman8-puppet_interm.cer"

We use certificates from a custom CA. foreman8-puppet.cer is the server certificate for the new proxy foreman8-puppet.example.com, foreman8-puppet.key is the key and foreman8-puppet_interm.cer is the full chain.

Copy the tar foreman8-puppet.example.com-certs.tar to the new proxy, /root and extract it there:

[root@foreman8-puppet ~]# tar xvf foreman8-puppet.example.com-certs.tar

This extracts everything into /root/ssl-build/

As far as I know by now, I need to configure ssl for the foreman-proxy (where the main server connects to the proxy) and the puppet client (which the proxy uses to connect to the puppet api on the main server to upload reports, facts, etc.).

foreman-proxy:

[root@foreman8-puppet ~]# cd /etc/foreman-proxy/
[root@foreman8-puppet foreman-proxy]# cp ~/ssl-build/katello-default-ca.crt ssl_ca.pem
[root@foreman8-puppet foreman-proxy]# cp ~/ssl-build/foreman8-puppet.example.com/foreman8-puppet.example.com-foreman-proxy.crt ssl_cert.pem
[root@foreman8-puppet foreman-proxy]# cp ~/ssl-build/foreman8-puppet.example.com/foreman8-puppet.example.com-foreman-proxy.key ssl_key.pem
[root@foreman8-puppet foreman-proxy]# chmod 440 ssl_ca.pem ssl_key.pem
[root@foreman8-puppet foreman-proxy]# chmod 444 ssl_cert.pem
[root@foreman8-puppet foreman-proxy]# chgrp foreman-proxy ssl_*.pem
[root@foreman8-puppet foreman-proxy]# ls -l
total 20
-rw-r--r--. 1 root root             0 Jun  9 16:46 migration_state
drwxr-xr-x. 2 root root          4096 Jul 14 15:49 settings.d
-rw-r-----. 1 root foreman-proxy 3968 Jun  9 16:46 settings.yml
-r--r-----. 1 root foreman-proxy 2524 Jul 14 12:53 ssl_ca.pem
-r--r--r--. 1 root foreman-proxy 3004 Jul 14 12:09 ssl_cert.pem
-r--r-----. 1 root foreman-proxy 3272 Jul 14 12:10 ssl_key.pem

Puppet client:

[root@foreman8-puppet ~]# mkdir -p /etc/pki/katello/puppet
[root@foreman8-puppet ~]# chmod 750 /etc/pki/katello/puppet/
[root@foreman8-puppet ~]# cd /etc/pki/katello/puppet/
[root@foreman8-puppet puppet]# cp ~/ssl-build/katello-server-ca.crt puppet_client_ca.crt
[root@foreman8-puppet puppet]# cp ~/ssl-build/foreman8-puppet.example.com/foreman8-puppet.example.com-puppet-client.crt puppet_client.crt
[root@foreman8-puppet puppet]# cp ~/ssl-build/foreman8-puppet.example.com/foreman8-puppet.example.com-puppet-client.key puppet_client.key
[root@foreman8-puppet puppet]# chmod 440 puppet_client*
[root@foreman8-puppet puppet]# chgrp -R puppet .
[root@foreman8-puppet puppet]# ls -al
total 24
drwxr-x---. 2 root puppet   84 Jul 14 18:22 .
drwxr-xr-x. 3 root root     20 Jul 14 18:17 ..
-r--r-----. 1 root puppet 6189 Jul 14 18:21 puppet_client_ca.crt
-r--r-----. 1 root puppet 8355 Jul 14 18:22 puppet_client.crt
-r--r-----. 1 root puppet 3272 Jul 14 18:22 puppet_client.key

This is the same organization files and directories like on the main server.

And point foreman-installer to the correct locations:

...
  --foreman-proxy-ssl-ca=/etc/foreman-proxy/ssl_ca.pem \
  --foreman-proxy-ssl-cert=/etc/foreman-proxy/ssl_cert.pem \
  --foreman-proxy-ssl-key=/etc/foreman-proxy/ssl_key.pem \
  --puppet-server-foreman-ssl-ca=/etc/pki/katello/puppet/puppet_client_ca.crt \
  --puppet-server-foreman-ssl-cert=/etc/pki/katello/puppet/puppet_client.crt \
  --puppet-server-foreman-ssl-key=/etc/pki/katello/puppet/puppet_client.key \
...

Now, basic connections seem to work. I can register the proxy in the main server and the proxy seems to forward facts and reports to the main server.

Looking at the other ssl configuration for foreman-proxy on the main server, I can see that it’s different certs here:

    --foreman-proxy-foreman-ssl-ca  SSL CA used to verify connections when accessing the Foreman API.
                                  When not specified, the ssl_ca is used instead. (current: "/etc/foreman-proxy/foreman_ssl_ca.pem")
    --foreman-proxy-foreman-ssl-cert  SSL client certificate used when accessing the Foreman API
                                  When not specified, the ssl_cert is used instead. (current: "/etc/foreman-proxy/foreman_ssl_cert.pem")
    --foreman-proxy-foreman-ssl-key  Corresponding key to a foreman_ssl_cert certificate
                                  When not specified, the ssl_key is used instead. (current: "/etc/foreman-proxy/foreman_ssl_key.pem")

I didn’t set these on my new proxy, thus it’s using the ssl_*.pem files instead. It’s unclear to me if and where the foreman-proxy uses those…

I have put my current “solution” into a tutorial: Installing a puppet smart proxy against a katello main server

Seems to work for me so far. I haven’t seen any issues.

It would be great, if foreman-installer was able to handle this and accept the certs-tar from katello…