Foreman puppet smartproxy certificate error

Support
, ,
1

Problem:
Puppet clients not registering to foreman proxy.
Expected outcome:
a regular puppet run.
Foreman and Proxy versions:
3.7.0
Foreman and Proxy plugin versions:
3.7.0
Distribution and version:
OL8
Other relevant data:

Hi everyone, we are moving away from a foreman with katello + proxy instance to a new foreman + proxy infra.

Im having an error registering puppet hosts to the proxy, but no error when registering directly to the main foreman server.
the eror says :

[root@er vagrant]# puppet agent -t
Error: CA certificate is missing from the server
Error: CA certificate is missing from the server

I can wget the https://foremanproxy00.#########:8140/puppet-ca/v1 but it says: ERROR: The certificate of ‘foremanproxy00.#######’ is not trusted.
ERROR: The certificate of ‘foremanproxy00.#######’ hasn’t got a known issuer.

on the proxy side we used (replacing the values):

puppet ssl bootstrap --server foreman.example.com

then

foreman-installer \
 --no-enable-foreman \
 --no-enable-foreman-plugin-puppet \
 --no-enable-foreman-cli \
 --no-enable-foreman-cli-puppet \
 --enable-puppet \
 --puppet-server-ca=false \
 --puppet-server-foreman-url=https://foreman.example.com \
 --enable-foreman-proxy \
 --foreman-proxy-puppetca=false \
 --foreman-proxy-foreman-base-url=https://foreman.example.com \
 --foreman-proxy-trusted-hosts=foreman.example.com \
 --foreman-proxy-oauth-consumer-key=<key here> \
 --foreman-proxy-oauth-consumer-secret=<secret here>

without errors and registered correctly

so, what could be the problem?

2

I think you are missing “–puppet-ca-server foreman.example.com” in your installer (not sure what that does in particular, but was needed for our smart-proxies). Also, I think your hosts are trying to use foremanproxy00 as puppet-ca (which it explicitly is not), so you need to set

ca_server = foreman.example.com

in the main section of /etc/puppetlabs/puppet/puppet.conf.
For systems that already tried to enroll as puppet client, you will probably need to remove Puppet’s ssl dir (/etc/puppetlabs/puppet/ssl/ by default) and re-do the ssl bootstrap either using puppet ssl bootstrap or automatically with a puppet run.

3

Thanks for the reply, so added –puppet-ca-server with the correct value and run the installer again. Still no luck.

We are looking to get the clients to register directly to the Foreman’s puppet proxy which is on a dmz, the foreman main server is not reachable to the clients.

4

Since it’s DMZ and the hosts may not communicate directly with the main Foreman instance, you have two options.

  1. The “clean” solution. Make your smart-proxy a seperate puppet ca. Since (as far as I know, I only have experience with Katello) Foreman without Katello uses the Puppet Certs for communication, you will have to store the certs the smart-proxy uses for communication with Foreman somewhere else and change the settings for that with foreman-installer. That should be --foreman-proxy-foreman-ssl-ca, --foreman-proxy-foreman-ssl-cert, --foreman-proxy-foreman-ssl-key, --foreman-proxy-ssl-ca, --foreman-proxy-ssl-cert and --foreman-proxy-ssl-key. In case you use PuppetDB, note that this will break communication with PuppetDB.
  2. The dirty hack. You can use nginx with nginx-mod-stream to reverse-proxy the PuppetCA from your DMZ. This will require you fake DNS names in the DMZ to match your reverse-proxy with foreman.example.com or update the Certs on your main Foreman instance to include an alt name for the reverse proxy. I would recommend against this if you can avoid it, but here is a nginx config file that works if you need to do that:
stream {
    server {
        listen     8140;
        proxy_pass foreman.example.com:8140;
    }
}

This needs to be outside of any nginx http section. Then you can point your clients to that server as CA and it should work.

I don’t know of any other ways to make this work. If possible, I would suggest you go with option 1 (for the obvious reason that it requires less dirty hacks).

5

I setup new testing vms with Foreman’s Forklift for quick testing, so I have 2 vms, one foreman server and one vm that would like to act as a proxy.
I copied the certificates from foreman to the proxy and executed the installer with -foreman-proxy-foreman-ssl-ca, --foreman-proxy-foreman-ssl-cert, --foreman-proxy-foreman-ssl-key, and the installer goes through without error. Here’s the installer opts:
foreman-installer
–no-enable-foreman
–no-enable-foreman-plugin-puppet
–no-enable-foreman-cli
–no-enable-foreman-cli-puppet
–enable-puppet
–puppet-server-ca=false
–puppet-server-foreman-url=https://foreman02.vms.nyx
–enable-foreman-proxy
–foreman-proxy-puppetca=true
–foreman-proxy-foreman-base-url=https://foreman02.vms.nyx
–foreman-proxy-trusted-hosts=foreman02.vms.nyx
–foreman-proxy-oauth-consumer-key=AdjKpxgiWbThWGn3kCvGpd3Mg2Zzs6zV
–foreman-proxy-oauth-consumer-secret=tJYpjmRjV6GZ2Vqx7mHkQdakKkcJELgR
–foreman-proxy-puppet-ssl-ca=/etc/puppetlabs/puppet/ssl2/ca.pem
–foreman-proxy-puppet-ssl-cert=/etc/puppetlabs/puppet/ssl2/cert-foreman02.vms.nyx.pem
–foreman-proxy-puppet-ssl-key=/etc/puppetlabs/puppet/ssl2/private_key-foreman02.vms.nyx.pem

if I add in the proxy installer --foreman-proxy-ssl-ca, --foreman-proxy-ssl-cert and --foreman-proxy-ssl-key using the same ca,cert and key I get a

/Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[proxy.vms.nyx]: Error making PUT request to https://foreman02.vms.nyx/api/v2/smart_proxies/2/refresh: Unable to communicate with the proxy: ERF12-2530 [ProxyAPI::ProxyException]: Unable to detect features ([RestClient::SSLCertificateNotVerified]: SSL_connect returned=1 errno=0 state=error: certificate verify failed (Hostname mismatch)) for proxy https://proxy.vms.nyx:8443/v2/features Please check the proxy is configured and running on the host.

any thoughts?

6

It sounds like you are using the Foreman server’s key and cert for the proxy. If you use the certs for foreman02 to run a service (in this case the smart-proxy) foreman-proxy02 (or whatever it’s call), that’s not going to work.
You will need to generate dedicated ones for the proxy (through Puppet) and use those for --foreman-proxy-ssl-cert and --foreman-proxy-ssl-key. My suggestion would be:

  • puppet ssl bootstrap on the smart-proxy
  • (depending on if you have auto-sign set up) sign the certificates on the main Foreman server with puppetserver ca sign --certname <proxy fqdn>
  • cp /etc/puppetlabs/puppet/ssl/*.pem /etc/foreman-proxy/
  • rm -rf /etc/puppetlabs/puppet/ssl/*
  • foreman-installer --foreman-proxy-ssl-ca /etc/foreman-proxy/ssl_ca.pem --foreman-proxy-ssl-cert /etc/foreman-proxy/client_cert.pem --foreman-proxy-ssl-key /etc/foreman-proxy/client_key.pem

This should do the trick, I would assume.

7

@areyus well, that’s exactly what I did on the first place, and clients registering to proxy are giving: Error: CA certificate is missing from the server
tried using foreman’s ssl certs on the proxy, same error on the clients.
now Im doing :

  • rm -rf /etc/puppetlabs/puppet/ssl/* to start from scratch
  • puppet ssl bootstrap --server foreman02.vms.nyx
  • signed the proxy at the main foreman server.
  • use newly proxy’s puppet ssl certs at the foreman-installer at the proxy
    all returns sucess, but, trying to register a new client to the proxy gives: Error: CA certificate is missing from the server
8

Just looked again at your installer command, you have –puppet-server-ca=false, so there is no CA set up on your smart-proxy despite the smart-proxy feature for puppet-ca being enabled. If you followed all the steps I mentioned above, foreman-installer –puppet-server-ca=true should fix this.

9

hmmm, its giving:
2023-09-05 11:13:55 [ERROR ] [configure] ‘/opt/puppetlabs/bin/puppetserver ca setup’ returned 1 instead of one of [0]
2023-09-05 11:13:55 [ERROR ] [configure] /Stage[main]/Puppet::Server::Config/Exec[puppet_server_config-generate_ca_cert]/returns: change from ‘notrun’ to [‘0’] failed: ‘/opt/puppetlabs/bin/puppetserver ca setup’ returned 1 instead of one of [0]

I also tried removing the ssl certs, bootstrap, sign the cert, copy the .pem, run the installer again, with same results.

10

Not sure if you missed mentioning it or if you actually skipped/missed that part both times, but you need to remove the Puppet SSL certs after copying them to another location. As long as the certs signed by the Foreman server’s Puppet CA are present, you cannot setup a new CA on your proxy.

If that’s not the case or it’s still failing to setup the CA, try to run puppetserver ca setup manually to see what the actual error is.

1 Like
11

Ok, now that part gets better, I followed that step and had an error which got solved using --foreman-server-ssl-crl .
after that the installer runs ok, a client can send a certificate request against the proxy, but, from the proxy, when I execute puppetserver ca list I get:

Error: Failed connecting to https://foreman02.vms.nyx:8140/puppet-ca/v1/certificate_statuses/any_key?state=requested
Root cause: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)

12

Looks like Puppet on your smart-proxy still thinks that foreman02 should be the CA server :thinking:
Maybe that’s a leftover bit from a former foreman-installer run, but foreman-installer --reset-puppet-ca-server should fix that by defaulting the puppet-ca-server option back to undef and unsetting ca_server in puppet.conf

13

Thanks for the reply, seems we are closer.
Tried that but keeps the same error as before trying to connect to the main foreman server.

14

Does puppet.conf still contain the ca_server setting?
Might also be worth checking if server is set and actually points to the smart-proxy. I think those are the only settings that should affect where puppet commands try to connect to.
If ca_server is unset (as I would expect now) and server points to foreman02, there should be an installer option to fix that to.

15

ok, tried on a new proxy system from scratch and the puppet part seems to accept the hosts now, but they are not shown at foreman’s UI.
Systems registering directly to the main foreman are ok as expected.

We have a foreman with katello, where systems registeing to the proxy with puppet are shown on the main foreman. Is there a way to do so?

16

The systems should show up through fact upload, which the ENC script should do automatically. Do you use Foreman as Puppet ENC and if so, is that part working?
Maybe the hosts are also just listed without organization/location. In case you have not checked that already. When using Katello, I think subscription-manager takes care of that somewhat automagically, but for plain Foreman afaik there’s some custom facts you have to set on the host for automatic assignment to take place.

17

Thanks for your reply!
Systems registering directly to foreman are shown ok .
Systems registering through the proxy are not.

No ENC has been changed/touched. Systems are not shown, not that they do not have Organization/Location. that’s not the problem. mmm what else could I look ?

18

Do you see any fact uploads for those clients in Foreman’s production.log?
If not, are there any errors in the smart-proxies puppetserver log?
Just poking at things to look at, no idea why it would not work.