Correct configuration of smart proxy with puppet ca forwarding?

Problem:

When register hosts i have a few issues, one of which is that when puppet is ran for the first time it will generate a signing request, but this is stuck on the smart proxy as its not relaying this to the main foreman server.

Expected outcome:

I would like the smart proxy to accept signing requests from clients, but it should relay this to the main foreman server which in turn will either autosign or get a manual sign, and then that would relay the client cert back to the proxy and through that to the client.

Ive read 4.3.7 Puppet CA in the docs but im not exactly sure on what im reading there, Im just assuming this is supported, considering a standard hub and spoke design with proxies in segmented networks and all the main services in the hub. If i interpret the docs correctly then puppetca_http_api should be set in puppetca_http_api.yml, im sure if i use the correct installer flags this is configured by the installer? (currently its set to the smartproxy itself.)

Foreman and Proxy versions:

foreman-installer.noarch 1:3.13.1-1.el9
foreman-installer-katello.noarch 1:3.13.1-1.el9
foreman-proxy.noarch 3.13.1-1.el9
foreman-proxy-content.noarch 4.15.0-1.el9
katello-certs-tools.noarch 2.10.0-1.el9
katello-client-bootstrap.noarch 1.7.9-2.el9
katello-common.noarch 4.15.0-1.el9

Distribution and version:

RHEL 9.6

Other relevant data:

I have the following installer options that i think are relevant:

--foreman-proxy-puppetca
Enable Puppet CA feature (current: false)

--puppet-ca-server                                                    
Use a different ca server. Should be either a string with the location of the ca_server or 'false'. (current: "$main_foreman_server")

One of the most common confusions in the Foreman ecosystem i believe is that foreman-proxy is actually not a proxy. Foreman-Proxy is more of a “satellite” with delegated responsibilities, it usually does not actually proxy any things between the client and the main Foreman server.
The use-case you describe is not possible with foreman-proxy setups, and due to the way the Puppet CA and SSL in general work, for a rather good reason.
I had a similar requirement a few years back and found there are basically two options:

  1. Run a separate Puppet CA on the smart-proxy server
  2. Set up some kind of reverse proxy service for PuppetCA yourself

Option 1. is by far the easier one. In my case, this was not actually possible since it broke our PuppetDB setup. Back then, I ended up going with option 2, setting up a reverse proxy on a seperate system with nginx stream, relaying the Puppet CA queries to the Puppet CA on the main Foreman server. This requires quite some workarounds at a lot of places, though, since Puppet performes quite a lot of validations in the certificate request workflow. This included, among other things, that I needed to “fake” the CA servers DNS name via /etc/hosts on the agents, because Puppet requires that the DNS name you use to communicate with the CA matches the name in the Puppet CA certificate.

In general, I would recommend to simply use the Puppet CA on the main Foreman server, if that is at all possible. If not, consider running a seperate Puppet CA on the smart-proxy server.
If both are absolutely not possible, you can make it work with a lot of fiddling and manual configuration, but you will need to configure quite a few things yourself and live with very ugly workarounds.