Setting up deployment to an isolated network

Problem:
I am trying to setup deployment to an isolated network (DMZ). Due to the restrictions applied by the network design, no connections from that network can be made directly to the rest of the datacenter, where or main Foreman server resides.
The setup currently looks like this:

  • Main Foreman server with some smart-proxies are deployed to the main datacenter network and server clients there
  • I have to multihomed smart-proxy servers deployed to a dedicated network layer that connects the DMZ to the main datacenter. One interface facing the DMZ, one facing the main network.
  • With some hacks (more than I liked, but it is what it is), I got deployment, content management and remote execution to work.
  • I am now stuck on setting up Puppet, since nodes in the DMZ are not able to connect to the main Foreman server where our Puppet CA is running.

The things I tried or at least considered by now are:

  • Setting up a new PuppetCA on one of the smart-proxies. After some fiddeling, this worked for basic Puppet stuff, but it breaks connection to the PuppetDB since the new Puppet masters need to use the new CA.
  • Setting up a second PuppetDB to solve the problems from above: I considered that, but foreman-puppetdb only supports one PuppetDB and I am afraid of other this breaking if I set up another one the plugin does not know about.
  • I considered trying to make the new PuppetCA a subCA of the main CA, but this does not seem to be supported by Puppet (anymore). I remember there being docs for such a setup way back in the day, but those are not there anymore and I am utterly lost if the external CA docs is what I should be looking at or if that would only complicate things further.
  • Proxying Puppet CA requests to the main PuppetCA. I set up a nginx server in stream mode to proxy requests to the Puppet CA on our main Foreman server. When I just set that server as ca_server in puppet.conf, I got hostname/certname missmatch errors (no big surprise, I guess). I then tried setting this up properly via proxy config, setting the proxy server as http_proxy in puppet.conf, but this results in connection errors I cannot debug since neither the puppetserver running the CA nor nginx log anything about that connection. I can see traffic going back and forth with TCPDump, but thats as far as I got debugging this.

Does anyone here have some experience with setups like this and can give some advice? If not, I will likely try and go with the “second PuppetDB” approach and just hope this will not break anything in the long run.

Expected outcome:
Puppet and PuppetDB connections working with smart-proxies for an isolated network.

Foreman and Proxy versions:
3.2.1

Foreman and Proxy plugin versions:
Plugins:

foreman-tasks 6.0.1
foreman_expire_hosts 7.0.4
foreman_hooks 0.3.17
foreman_puppet 3.0.7
foreman_remote_execution 6.0.0
foreman_scc_manager 1.8.20
foreman_snapshot_management 2.0.1
foreman_templates 9.1.0
katello 4.4.1
puppetdb_foreman 5.0.0

Distribution and version:
RHEL7

Other relevant data:

After fiddling around a bit more, the Proxy solution via nginx does work if I set the main Foreman server as puppetCA, remove the proxy config from the puppet.conf and add foreman.example.com (main server) as an alias for the nginx server to /etc/hosts.

It’s working, but this still feels like some super dirty hack I would rather avoid if possible. So any hints on how to do this better are greatly appreciated.