Foreman Proxy and Puppet CA

Problem:
I have a Foreman proxy running in a network that does not have access to the Foreman server. The proxy itself has access, but the rest of the network does not. Content and templates are being served properly. However, when a puppet client is pointed to the server, I get the following error:

Error: CRL is missing from the server
Error: CRL is missing from the server

The proxy has been installed with the following options:

foreman-installer \
--scenario foreman-proxy-content \
--certs-tar-file  "/root/$proxy-certs.tar" \
--foreman-proxy-register-in-foreman  "true" \
--foreman-proxy-foreman-base-url "https://$foreman" \
--foreman-proxy-trusted-hosts  "$foreman" \
--foreman-proxy-trusted-hosts "$proxy" \
--foreman-proxy-oauth-consumer-key "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
--foreman-proxy-oauth-consumer-secret "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" \
--enable-puppet \
--foreman-proxy-puppet true \
--foreman-proxy-puppetca true \
--foreman-proxy-registered-proxy-url "https://$proxy:9090" \
--foreman-proxy-template-url "https://$proxy:9090" \
--foreman-proxy-templates-listen-on https \
--foreman-proxy-content-enable-ansible false \
--foreman-proxy-content-enable-python false \
--foreman-proxy-plugin-remote-execution-script-cockpit-integration false \
--puppet-ca-server "$foreman" \
--enable-foreman-proxy-plugin-remote-execution-script \
--foreman-proxy-puppet-url "https://$foreman:8140" \
--foreman-proxy-manage-puppet-group true \
--puppet-server true \
--puppet-server-foreman-url "https://$foreman/" \
--puppet-server-ca-crl-sync true \
--puppet-server-ca false

Expected outcome:

I would expect the proxy would provide the CA data from the main Foreman server. Am I missing an option? Have I misunderstood an option and specified it incorrectly?

Foreman and Proxy versions:
Foreman:
foreman-3.13.0-1.el9.noarch
foreman-proxy-3.13.0-1.el9.noarch

Proxy:
foreman-proxy-3.13.0-1.el9.noarch

Foreman and Proxy plugin versions:
rubygem-smart_proxy_container_gateway-3.1.1-1.fm3_13.el9.noarch
rubygem-smart_proxy_pulp-3.4.0-1.fm3_13.el9.noarch
rubygem-smart_proxy_dynflow-0.9.3-1.fm3_12.el9.noarch
rubygem-smart_proxy_remote_execution_ssh-0.11.4-1.fm3_13.el9.noarch

Distribution and version:
Oracle Linux 9.5

Other relevant data:
This is a snippet of the agent running with debugging

Debug: Creating new connection for https://proxy.itlab.mydomain.com:8140
Debug: Starting connection for https://proxy.itlab.mydomain.com:8140
opening connection to proxy.itlab.mydomain.com:8140...
opened
starting SSL for proxy.itlab.mydomain.com:8140...
SSL established, protocol: TLSv1.3, cipher: TLS_AES_128_GCM_SHA256
Debug: Using TLSv1.3 with cipher TLS_AES_128_GCM_SHA256
Debug: Caching connection for https://proxy.itlab.mydomain.com:8140
Debug: Resolved service 'ca' to https://proxy.itlab.mydomain.com:8140/puppet-ca/v1
Debug: Using cached connection for https://proxy.itlab.mydomain.com:8140
<- "GET /puppet-ca/v1/certificate_revocation_list/ca HTTP/1.1\r\nX-Puppet-Version: 8.10.0\r\nUser-Agent: Puppet/8.10.0 Ruby/3.2.5-p208 (x86_64-linux)\r\nAccept: text/plain\r\nAccept-Encoding: gzip;q=1.0,deflate;q=0.6,identity;q=0.3\r\nHost: proxy.itlab.mydomain.com:8140\r\n\r\n"
-> "HTTP/1.1 404 Not Found\r\n"

You have indeed misunderstood something here. There is no way to proxy the Puppet CA with foreman-proxy. The --foreman-proxy-puppet-ca option enabled the foreman-proxy capability to manage a Puppet CA service, but you disabled the CA service via --puppet-server-ca false, so there is no CA the clients can connect to.

The intended way how you set things up is to run a dedicated Puppet CA on the smart-proxy server and have that manage certificates of the isolated hosts. This takes a bit of fiddling with installer options, since foreman-proxy (the service) needs to use and trust certificates from the main Puppet CA, but the Puppet Server needs to use it’s own local CA: It has been a few years since I last tried to set this up, so I cannot give you any specific directions, but when setup correctly I can confirm it works. The only thing I could not get to work with this approach is connectiong to that to a central PuppetDB instance because of certificate issues.

Hope this helps
Regards