Error trying to setup foreman-proxy with central foreman master

[flechoide]

Problem:
I have an error when i go with puppet agent -t on a client :

[root@puppetclient ~]# puppet agent -t
Error: CA certificate is missing from the server
Error: Could not run: CA certificate is missing from the server

Expected outcome:
The expected outcome would be the generation of certificates and the run of puppet agent

Foreman and Proxy versions:
[root@puppetclient ~]# cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)
[root@foremanmaster ~]# rpm -qa | egrep -i “foreman|puppet”
tfm-rubygem-hammer_cli_foreman-0.19.6-1.el7.noarch
foreman-installer-1.24.2-1.el7.noarch
puppetserver-6.9.0-1.el7.noarch
foreman-proxy-1.24.2-1.el7.noarch
puppet6-release-6.0.0-5.el7.noarch
puppet-agent-6.13.0-1.el7.x86_64
foreman-cli-1.24.2-1.el7.noarch
foreman-selinux-1.24.2-1.el7.noarch
foreman-1.24.2-1.el7.noarch
foreman-debug-1.24.2-1.el7.noarch
puppet-agent-oauth-0.5.1-3.el7.noarch
foreman-release-1.24.2-1.el7.noarch
foreman-release-scl-7-2.el7.noarch
foreman-postgresql-1.24.2-1.el7.noarch
[root@foremanmaster ~]#

Foreman and Proxy plugin versions:
see above

Distribution and version:

[root@puppetclient ~]# cat /etc/centos-release
CentOS Linux release 7.7.1908 (Core)

[flechoide]

I had to split that with 2 messages because the form shows an error about links.

Other relevant data:
What Im trying to implement here is a small lab using static ip approach since I cannot use dhcp later in real environments , where I have 1 foreman master with everything , 1 foremanproxy what should be able to conect to the central foreman master, 1 puppet client that should be able to conect to central foremanmaster using the foremanproxy.

foremanmaster :
192.168.1.184/24

foremanproxy
192.168.1.185/24
192.168.2.185/24

puppetclient
192.168.2.186/24

So I created 3 centos virtualm machines and instaled the apropiate stuff with the following repos from the quickstart :
rpm -iUhv https://yum.puppet.com/puppet6-release-el-7.noarch.rpm
yum -y install epel-release
yum -y install http://yum.theforeman.org/releases/1.24/el7/x86_64/foreman-release.rpm
yum -y install foreman-installer

Everything runs ok :

[root@foremanmaster ~]# foreman-installer
Preparing installation Done
Success!

• Foreman is running at https://foremanmaster.mydomain.com
  Initial credentials are admin / ----cENmYiQyg----
• Foreman Proxy is running at https://foremanmaster.mydomain.com8443
• Puppetmaster is running at port 8140
  The full log is at /var/log/foreman-installer/foreman.log
  [root@foremanmaster ~]#

[flechoide]

Then following the documentation I create the following certs and I copy them to the proxy :
[root@foremanmaster ~]# puppetserver ca generate --certname foremanproxy.mydomain.com

I then create on the proxy the directory and copy the files astated at the documentation :
mkdir -p /etc/puppetlabs/puppet/ssl/{private_keys,certs}

scp /etc/puppetlabs/puppet/ssl/private_keys/foremanproxy.mydomain.com.pem foremanproxy:/etc/puppetlabs/puppet/ssl/private_keys/foremanproxy.mydomain.com.pem
scp /etc/puppetlabs/puppet/ssl/certs/foremanproxy.mydomain.com.pem foremanproxy:/etc/puppetlabs/puppet/ssl/certs/foremanproxy.mydomain.com.pem
scp /etc/puppetlabs/puppet/ssl/certs/ca.pem foremanproxy:/etc/puppetlabs/puppet/ssl/certs/ca.pem

Then I go to foremanproxy and install as it says at the documentation :
foreman-installer
–no-enable-foreman
–no-enable-foreman-cli
–no-enable-foreman-plugin-bootdisk
–no-enable-foreman-plugin-setup
–enable-puppet
–puppet-server-ca=false
–puppet-server-foreman-url=https://foremanmaster.mydomain.com
–enable-foreman-proxy
–foreman-proxy-puppetca=false
–foreman-proxy-tftp=false
–foreman-proxy-foreman-base-url=https://foremanmaster.mydomain.com
–foreman-proxy-trusted-hosts=foremanmaster.mydomain.com
–foreman-proxy-oauth-consumer-key=…2nnzWdKpHGWh68BK…
–foreman-proxy-oauth-consumer-secret=…4zDKG5d45CRTr…

[flechoide]

Preparing installation Done
Success!

• Foreman Proxy is running at https://foremanproxy.mydomain.com:8443
• Puppetmaster is running at port 8140
  The full log is at /var/log/foreman-installer/foreman.log
  [root@foremanproxy ~]#

Then I install the puppet agent package at the puppet client and config the client puppet.conf :
[root@puppetclient ~]# egrep -v “^$|^#” /etc/puppetlabs/puppet/puppet.conf
server = foremanproxy.mydomain.com
[root@puppetclient ~]#

But it fails :
[root@puppetclient ~]# puppet agent -t
Error: CA certificate is missing from the server
Error: Could not run: CA certificate is missing from the server

It seems like the proxy doesnt have an apropiate CA but I copied that from the master :
[root@foremanproxy ~]# ls -larth /etc/puppetlabs/puppet/ssl/certs
total 8.0K
-rw-r–r-- 1 puppet puppet 2.0K Mar 5 04:18 foremanproxy.mydomain.com.pem
-rw-r–r-- 1 puppet puppet 3.8K Mar 5 04:18 ca.pem
drwxr-xr-x 2 puppet puppet 55 Mar 5 04:18 .
drwxrwx–x 7 puppet puppet 117 Mar 5 04:26 …

And everything is up :
[root@foremanproxy ~]# netstat -netapl | grep LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 0 23029 924/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 23610 1343/master
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 0 16974 1/systemd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 22780 922/sshd
tcp6 0 0 ::1:631 :::* LISTEN 0 23028 924/cupsd
tcp6 0 0 ::1:25 :::* LISTEN 0 23611 1343/master
tcp6 0 0 :::8443 :::* LISTEN 987 39326 10207/ruby
tcp6 0 0 :::8140 :::* LISTEN 986 38380 9925/java
tcp6 0 0 :::111 :::* LISTEN 0 16976 1/systemd
tcp6 0 0 :::22 :::* LISTEN 0 22782 922/sshd
[root@foremanproxy ~]#

I see at foremanproxy that it doesnt have a certificate from /var/log/puppetlabs/puppetserver/puppetserver-access.log :
192.168.2.186 - - [05/Mar/2020:05:12:50 -0800] “GET /puppet-ca/v1/certificate/ca HTTP/1.1” 404 0 “-” “Puppet/6.13.0 Ruby/2.5.7-p206 (x86_64-linux)” 1 - -
192.168.1.185 - - [05/Mar/2020:05:13:09 -0800] “GET /puppet-ca/v1/certificate_revocation_list/ca HTTP/1.1” 404 0 “-” “Puppet/6.13.0 Ruby/2.5.7-p206 (x86_64-linux)” 1 - -

So what Im missing or Im doing wrong in order to get the foremanproxy comunicate with the foreman master ¿?

[flechoide]

Im not sure if thats an accepted error but may be related :

[root@foremanmaster ~]# puppetserver ca generate --certname foremanproxy.mydomain.com
Successfully saved private key for foremanproxy.mydomain.com to /etc/puppetlabs/puppet/ssl/private_keys/foremanproxy.mydomain.com.pem
Successfully saved public key for foremanproxy.mydomain.com to /etc/puppetlabs/puppet/ssl/public_keys/foremanproxy.mydomain.com.pem
Successfully submitted certificate request for foremanproxy.mydomain.com
Error:
Signed certificate foremanproxy.mydomain.com could not be found on the CA
Successfully signed certificate request for foremanproxy.mydomain.comSuccessfully saved certificate for foremanproxy.mydomain.com to /etc/puppetlabs/puppet/ssl/certs/foremanproxy.mydomain.com.pem
[root@foremanmaster ~]# puppetserver ca list --all
Signed Certificates:
foremanmaster.mydomain.com (SHA256) 5D:7C:EB:95:25:5D:DC:99:7D:E6:17:00:96:8B:72:84:C4:E2:EF:2D:15:D8:13:C6:1A:DC:DE:50:14:F1:BF:36 alt names: [DNS:puppet, DNS:foremanmaster.mydomain.com]
foremanproxy.mydomain.com (SHA256) 5E:D4:02:B5:09:9E:8F:96:D7:CC:56:B1:BA:55:57:3E:A5:6D:C1:49:0F:DF:DA:03:A7:23:82:1E:DD:83:FA:ED

[jjack]

Your issues seem related to your previous posting:

Multi proxy setup for puppet over different networks

Hi and welcome to the forum Foreman Smart-Proxies are designed for the use-case you are describing, but they do not work like a classical proxy. The architecture with smart-proxy is essentially the following: You have your Foreman Server as a central management instance, where you assign things like Puppet environment, classes and parameters to send via Puppet ENC, etc. On each separated subnet you deploy a smart-proxy server. For Puppet functionality, this server acts as a st…

where you are thinking that the “Foreman Smart-Proxy for Puppet” is a proxy for Puppet clients which then speaks to a Puppet Master.

I see you have a puppet client pointing at a Foreman Proxy as if the Foreman Proxy machine is running a puppetserver process and is acting as a Puppet Master.

[root@puppetclient ~]# egrep -v “^$|^#” /etc/puppetlabs/puppet/puppet.conf
server = foremanproxy.mydomain.com

Is your foremanproxy.mydomain.com running as a Puppet Master? I could be wrong but in my experience the Foreman Proxy for Puppet is a proxy for the Puppet Master to speak with the Foreman instance so that Foreman can know the Puppet Environments & Puppet Classes that the Puppet Master knows.

And I think @areyus answered this in:

@areyus

• On each separated subnet you deploy a smart-proxy server. For Puppet functionality, this server acts as a standalone puppet-master. It communicates with the central Foreman for ENC information, but is otherwise it’s own, standalone Puppetserver.
• Some Foreman Smart-Proxy features act differently, more in the way you would assume by the name of “proxy” (essentially routing the requests through to Foreman), but for Puppet that is not the case.

[flechoide]

Hello jjack, first let me thank for your time replying, you are right , the question is related to my previous post .

Actually what I think I did is setup an standalone foreman server which i call master, and a second foreman server with the certificates. You are right I have a puppet client pointing to the foremanproxy server which I thought was acting as a proxy server for some purposes, from the documentation :

3.2.3 Installation Scenarios

The Foreman installer can accommodate more complex, multi-host setups when supplied with appropriate parameters.

Setting up Foreman with external Puppet masters

Using the scenarios outlined below, a simple scale-out setup can be created as follows:

1. On the Foreman host, run a complete foreman-installer all-in-one installation to provide Foreman, a Puppet master and smart proxy. This will be the Puppet CA.

For each Puppet master:

1. Generate a new certificate following the steps in the SSL CA section and transfer it to the new Puppet master host
2. Run the standalone Puppet master installation as detailed below
   Each Puppet master will register with Foreman as a smart proxy, while the instance running on the Foreman host itself will act as a central Puppet CA. These can be selected while adding new hosts or host groups.

So far I understood that this is the scenario I was looking for, in fact when I go to the web console to infraestructure > smart proxy I see the server I call foremanmaster and the server I call foremanproxy
The foreman master with this characteristics : HTTPBoot, Logs, Puppet, Puppet CA and TFTP
The foreman proxy with this characteristics : Logs and Puppet

So I understand that foremanproxy.muydomain.com is acting as a puppet master yes , Im not able to understand what you are saying , so where is my error ? I think we are speaking the same, the server foremanproxy is a standalone puppet server that uses via foreman integration the info provided by the server foreman master in order to get the ENC info (clasification from puppet for vars and groups etc)

In fact the port 8140 is up :
root@foremanproxy ~]# netstat -netapl | grep LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 0 23029 924/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 0 23610 1343/master
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 0 16974 1/systemd
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 22780 922/sshd
tcp6 0 0 ::1:631 :::* LISTEN 0 23028 924/cupsd
tcp6 0 0 ::1:25 :::* LISTEN 0 23611 1343/master
tcp6 0 0 :::8443 :::* LISTEN 987 39326 10207/ruby
tcp6 0 0 :::8140 :::* LISTEN 986 38380 9925/java
tcp6 0 0 :::111 :::* LISTEN 0 16976 1/systemd
tcp6 0 0 :::22 :::* LISTEN 0 22782 922/sshd
[root@foremanproxy ~]#

[flechoide]

I think I forget that part, also I dont understand what the ssl bootstrap is and how ti works :

For each additional Smart Proxy:

1. Bootstrap certificates
2. Run the standalone installation as detailed below

Note This relies on the puppet ssl subcommand introduced in Puppet 6. Prior to Puppet 6 there was no separate command and it required manual work.

Assuming the Puppetserver with CA is on foreman.example.com , the following command can be run:

puppet ssl bootstrap --server foreman.example.com

[flechoide]

I think the problem here is that foreman doesnt handle out of the box the implementation of an external puppet CA , someone has managed to implement this kind of arqhitechtures ?

[ekohl]

I’ve played with such a deployment. This goes a step further and also deploys Puppetserver on a separate server without the Foreman Proxy. Right now that requires some additional work that is not present there so for now I’d recommend to keep Foreman Proxy and Puppetserver on the same machine:

theforeman/forklift/blob/master/pipelines/foreman_split_puppet.yml

[flechoide]

I apologize since I realize now that I messed up things because I read documentation from foreman 1.9 and foreman 1.24 and they are quite different since foreman 1.24 rely on puppet 6 .

@ekohl thanks for posting, whats that split ? makes me remember to my ansible master playbooks from my ansible environment . Ok I see that you have implemented an ansible playbook quite extensive , looks nice, these hosts , what they are ? :

• “{{ forklift_server_name }}” <-- that one has foreman and nothing else ?
• “{{ forklift_proxy_name }}” <-- this seems to be the puppet central master, that asumes almost every component , with the ca, foreman , proxy etc
• “{{ forklift_puppet_name }}” <-- I dont understand what is this EDIT : I see not that has puppet-server true so thats an isolated puppet server

I see you update the 3 servers, and place the repos as I do that manually . Later you implement the foreman-installer launcher with options, looks good to me but I dont get the whole strategy, also there are lots of options I didnt use from the main documentation

[ekohl]

@flechoide

I see you update the 3 servers, and place the repos as I do that manually . Later you implement the foreman-installer launcher with options, looks good to me but I dont get the whole strategy, also there are lots of options I didnt use from the main documentation

The strategy is to first install a puppetserver, then a foreman server and then a foreman-proxy all on separate hosts. This is probably overkill for most setups and the autosign setup that’s needed for provisioning doesn’t work since it relies on changing the autosign.conf file. That’s why I suggested to keep foreman-proxy and puppetserver on the same host. You’d need to configure the puppetca_token_whitelisting provider for that (which is more secure, but not the smoothest experience).

What you are trying to is an advanced use case and it’s expected that you dig in. It can be made to work and all the basic pieces are there, but it’s not really documented how to get them into the right place.

If you’re a Puppet user, you should know that the installer is written in Puppet and patches are very welcome. I’ll happily point you in the right direction.

[flechoide]

Hello, Im a puppet certified user buts been some time since I last went deep (we have an obsolete puppetserver 4 here and we want to implement a new (completely new) puppet 6 ), so Im doing my best to learn and remember and really apreciate the help .

I agree the 3 hosts strategy is overkill (no ofense intended here) and I can go with a server that has the puppetserver+ca+foreman proxy .

I see from the doc settings for smartproxy that I have to go with either token or hostname whitelisting provider, where can I find more info abouth that settings from foreman ? Im intuiting that you went with the strategy of 3 separate server because of using hostname_whitelisting ? where can I find more info or documentation ?

[flechoide]

Ok, I completely remade the process

foreman-installer
–enable-puppet
–puppet-server-foreman-url=https://foremanmaster.mydomain.lab
–enable-foreman-proxy
–foreman-proxy-puppetca=false
–foreman-proxy-tftp=false
–foreman-proxy-foreman-base-url=https://foremanmaster.mydomain.lab
–foreman-proxy-trusted-hosts=foremanmaster.mydomain.lab
–foreman-proxy-trusted-hosts=foremanproxy.mydomain.lab
–puppet-server-admin-api-whitelist=foremanmaster.mydomain.lab
–puppet-server-admin-api-whitelist=foremanproxy.mydomain.lab
–puppet-server-ca-client-whitelist=localhost
–puppet-server-ca-client-whitelist=foremanmaster.mydomain.lab
–puppet-server-ca-client-whitelist=foremanproxy.mydomain.lab
–foreman-proxy-puppetca-provider=puppetca_token_whitelisting

Then on the server “proxy” I bootstraped and signed from the master :
puppet ssl bootstrap --server foremanmaster.vlspr.home

Then I installed the foreman on to the “proxy” with just puppet without CA :
foreman-installer
–no-enable-foreman
–no-enable-foreman-cli
–enable-puppet
–puppet-server-ca=false
–puppet-server-foreman-url=https://foremanmaster.mydomain.lab
–enable-foreman-proxy
–foreman-proxy-puppetca=false
–foreman-proxy-tftp=false
–foreman-proxy-foreman-base-url=https://foremanmaster.mydomain.lab
–foreman-proxy-trusted-hosts=foremanmaster.mydomain.lab
–foreman-proxy-oauth-consumer-key=…
–foreman-proxy-oauth-consumer-secret=…

The problem is the same , when I try to use the client from the third machine it says :
[root@puppetclient ~]# puppet agent -t
Error: CA certificate is missing from the server
Error: Could not run: CA certificate is missing from the server

Then if I use the ca_server = foremanmaster.mydomain.lab it says it cannot reach the ca_server because is trying to acces the server directly

Im stuck here , not sure how to get the puppet client to synch with the puppetserver using the central CA without the central CA not being accesible by the client , maybe what Im trying doesnt exist ?

[flechoide]

Also I switched to puppetca_token_whitelisting at /etc/foreman-proxy/settings.d/puppetca.yml .
Changed the autosign to autosign = /usr/libexec/foreman-proxy/puppet_sign.rb
Changed at /etc/foreman-proxy/settings.d/puppetca_http_api.yml for http_api

Then on the server “proxy” I bootstraped and signed from the master :
puppet ssl bootstrap --server foremanmaster.vlspr.home

There is one aditional step Im missing that enables the puppetserver foremanproxy to comunicate with puppetserver foremanmaster (the one with the CA)

[minashvili]

Hello

I understand that a lot of time has passed, but I got such a mistake
The problem in trusted certificates Foreman-Proxy and PuppetServer can be made different

After the certificates are written and Foreman-Proxy installed

systemctl stop foreman-proxy.service puppetserver

Change the default paths for certificates (not use PuppetServer SSL)

/etc/foreman-proxy/settings.yml.

: ssl_ca_file: /etc/foreman-proxy/ca_form_main_foreman_server.pem
: ssl_certificate: /etc/foreman-proxy/crt_for_proxy_server.pem
: ssl_private_key: /etc/foreman-proxy/Key_for_Proxy_Server.pem

: trusted_hosts:

• Your_Main_Foreman_Server.
• Your_Foreman-Proxy

systemctl start foreman-proxy.service puppetserver

And in graphic interference Main_Foreman add a new proxy server

But after that, I have next a problem

What functionality can be expected from Foreman-Proxy ( Puppet, Puppet CA ) in the main Foreman-server( Puppet, Puppet CA )

Hello everyone =) I train installed Foreman-proxy( Puppet + Puppet CA ) I set up Foreman-Proxy(Puppet + Puppet CA) In an isolated network. Connected Puppet agents to the Foreman-proxy. And connected the Foreman-proxy to my main Foreman-server(Puppet + Puppet CA) And now I am trying to understand what functionality to expect from Foreman-Proxy in the graphical interface of my main Foreman-Server i can only manage CA ( Screenshot number 1 ) But I can not see how many servers in support fro…

Maybe my way is not true =D