Unable to install feature 'puppet' on smart proxy server

Problem:
Unable to install feature ‘puppet’ on smart proxy server

Expected outcome:
Smart proxy should get installed

Foreman and Proxy versions:
FOREMAN SERVER: sjprdsatapp01
[root@sjprdsatapp01 ~]# rpm -qa | grep -i foreman
foreman-ec2-3.7.0-1.el8.noarch
foreman-dynflow-sidekiq-3.7.0-1.el8.noarch
rubygem-foreman_discovery-22.0.4-1.fm3_7.el8.noarch
rubygem-foreman_remote_execution-10.0.7-1.fm3_7.el8.noarch
foreman-release-3.7.0-1.el8.noarch
rubygem-foreman-tasks-8.1.1-1.fm3_7.el8.noarch
foreman-postgresql-3.7.0-1.el8.noarch
rubygem-foreman_maintain-1.3.2-1.el8.noarch
foreman-vmware-3.7.0-1.el8.noarch
foreman-debug-3.7.0-1.el8.noarch
foreman-installer-katello-3.7.0-1.el8.noarch
rubygem-foreman_default_hostgroup-7.0.0-1.fm3_6.el8.noarch
foreman-proxy-3.7.0-1.el8.noarch
foreman-cli-3.7.0-1.el8.noarch
rubygem-hammer_cli_foreman_tasks-0.0.19-1.fm3_7.el8.noarch
foreman-service-3.7.0-1.el8.noarch
foreman-installer-3.7.0-1.el8.noarch
rubygem-hammer_cli_foreman_remote_execution-0.2.3-1.fm3_7.el8.noarch
rubygem-hammer_cli_foreman-3.7.0-1.el8.noarch
foreman-3.7.0-1.el8.noarch

Foreman proxy: sjpuppetm02
[root@sjpuppetm02 ~]# rpm -qa | grep -i foreman
foreman-proxy-3.7.0-1.el8.noarch
foreman-installer-3.7.0-1.el8.noarch
foreman-installer-katello-3.7.0-1.el8.noarch
foreman-debug-3.7.0-1.el8.noarch
rubygem-foreman_maintain-1.3.2-1.el8.noarch
foreman-proxy-content-4.9.1-1.el8.noarch

Foreman and Proxy plugin versions:

Distribution and version:

Other relevant data:

Environment:
foreman server: sjprdsatapp01
puppetca: sjpuppetm01

Need help to install smart proxy with ‘puppet’.
Is it necessary to install puppet on foreman server as well?
I am trying to setup an environment where only smart proxies will have puppet installed.
Foreman server will be used just to manage the smart proxies.

  1. I m trying to install smart proxy with puppet below options but getting below error:
# foreman-installer --scenario foreman-proxy-content --certs-tar-file "/root/sjpuppetm02.cadence.com-certs.tar" 
--foreman-proxy-register-in-foreman "true" --foreman-proxy-foreman-base-url "https://sjprdsatapp01.cadence.com" 
--foreman-proxy-trusted-hosts "sjprdsatapp01" --foreman-proxy-trusted-hosts "sjpuppetm02.cadence.com" 
--foreman-proxy-oauth-consumer-key "XXXXXXXXXXXXXXX" --foreman-proxy-oauth-consumer-secret "YYYYYYYYYYYYYYYYY" 
--foreman-proxy-puppet "true" --puppet-ca-server "sjpuppetm01.cadence.com" --puppet-server-ca "true" 
--foreman-proxy-puppetca "false" --enable-puppet --puppet-server "true" --puppet-server-foreman-url "https://sjprdsatapp01.cadence.com"



2023-10-09 05:34:34 [NOTICE] [root] Loading installer configuration. This will take some time.
2023-10-09 05:34:35 [NOTICE] [root] Running installer with log based terminal output at level NOTICE.
2023-10-09 05:34:35 [NOTICE] [root] Use -l to set the terminal output log level to ERROR, WARN, NOTICE, INFO, or DEBUG. See --full-help for definitions.
2023-10-09 05:34:37 [NOTICE] [configure] Starting system configuration.
2023-10-09 05:35:43 [NOTICE] [configure] 250 configuration steps out of 1542 steps complete.
2023-10-09 05:36:40 [NOTICE] [configure] 500 configuration steps out of 1545 steps complete.
2023-10-09 05:36:43 [NOTICE] [configure] 750 configuration steps out of 1551 steps complete.
2023-10-09 05:36:49 [NOTICE] [configure] 1000 configuration steps out of 1571 steps complete.
2023-10-09 05:37:00 [NOTICE] [configure] 1250 configuration steps out of 1571 steps complete.
2023-10-09 05:37:59 [NOTICE] [configure] 1500 configuration steps out of 1572 steps complete.
2023-10-09 05:38:06 [ERROR ] [configure] Proxy sjpuppetm02.cadence.com has failed to load one or more features (Puppet), check /var/log/foreman-proxy/proxy.log for configuration errors
2023-10-09 05:38:06 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[sjpuppetm02.cadence.com]/ensure: change from 'absent' to 'present' failed: Proxy sjpuppetm02.cadence.com has failed to load one or more features (Puppet), check /var/log/foreman-proxy/proxy.log for configuration errors
2023-10-09 05:38:07 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[sjpuppetm02.cadence.com]: Failed to call refresh: Proxy sjpuppetm02.cadence.com has failed to load one or more features (Puppet), check /var/log/foreman-proxy/proxy.log for configuration errors
2023-10-09 05:38:07 [ERROR ] [configure] /Stage[main]/Foreman_proxy::Register/Foreman_smartproxy[sjpuppetm02.cadence.com]: Proxy sjpuppetm02.cadence.com has failed to load one or more features (Puppet), check /var/log/foreman-proxy/proxy.log for configuration errors
2023-10-09 05:38:09 [NOTICE] [configure] System configuration has finished.

  There were errors detected during install.
  Please address the errors and re-run the installer to ensure the system is properly configured.
  Failing to do so is likely to result in broken functionality.

  The full log is at /var/log/foreman-installer/foreman-proxy-content.log
  1. Note that, sjpuppetm02 does get registered with foreman but feature ‘puppet’ does not get added.
    Attaching screenshot.

[root@sjpuppetm02 ~]# cat /var/log/foreman-proxy/proxy.log
2023-10-09T05:38:05  [I] Successfully initialized 'container_gateway'
2023-10-09T05:38:05  [I] Successfully initialized 'pulpcore'
2023-10-09T05:38:05  [I] Successfully initialized 'foreman_proxy'
2023-10-09T05:38:05  [I] Successfully initialized 'templates'
2023-10-09T05:38:05  [I] Successfully initialized 'puppet_proxy_puppet_api'
2023-10-09T05:38:05  [I] Successfully initialized 'puppet'
2023-10-09T05:38:05  [I] Successfully initialized 'logs'
2023-10-09T05:38:05  [I] Successfully initialized 'registration'
2023-10-09T05:38:05  [I] WEBrick 1.8.1
2023-10-09T05:38:05  [I] ruby 2.7.4 (2021-07-07) [x86_64-linux]
2023-10-09T05:38:05  [I]
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            0c:75:72:cb:32:a3:f7:8b:19:66:e2:47:20:fe:02:39:38:66:34:db
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C=US, ST=North Carolina, L=Raleigh, O=Katello, OU=SomeOrgUnit, CN=sjprdsatapp01.cadence.com
        Validity
            Not Before: Oct  2 12:08:30 2023 GMT
            Not After : Jan 18 12:08:32 2038 GMT
        Subject: C=US, ST=North Carolina, O=FOREMAN, OU=SMART_PROXY, CN=sjpuppetm02.cadence.com
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (4096 bit)
                Modulus:
                    00:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:07:
                    94:fd:b6:67:bb:e1:a1:82:d3:ef:aa:8a:c2:52:d3:
                    82:80:08:00:81:60:9a:d7:d2:33:83:23:f8:c3:88:
                    1d:fe:c6:b4:68:e6:d3:9a:58:d0:cd:ef:17:ad:31:
                    19:fd:c0:dc:30:a3:8e:4e:c8:73:b5:54:f8:2a:14:
                    f7:71:57:8f:51:50:c8:0a:46:dd:c9:3e:e7:10:97:
                    f6:cf:c4:a4:fb:94:16:62:ca:fb:e0:af:f4:17:a7:
                    c5:df:eb:2a:dc:ae:6a:e4:38:fc:90:83:cd:12:40:
                    ae:12:c9:bc:d6:59:39:c2:cf:d8:b7:29:a9:1e:01:
                    0b:ba:61:65:f3:50:0f:dc:9e:c7:10:59:a4:78:e0:
                    51:ab:b6:d8:fc:60:04:ae:c5:c8:d0:24:49:5e:ac:
                    12:8d:a6:ca:ad:43:52:68:89:25:f7:d3:f5:9a:ba:
                    c3:76:13:8b:0f:24:56:a1:1f:36:89:56:1f:e4:67:
                    55:c2:60:67:82:d6:1b:bc:bb:8c:b4:67:0f:55:6b:
                    8d:23:0f:32:f6:b2:0d:1e:e9:16:e2:79:e8:6a:f5:
                    9f:85:2f:e2:6d:92:93:76:af:7d:28:a8:a6:06:07:
                    8d:1c:c7:64:ad:a8:7d:d2:ea:15:5c:aa:7a:a5:48:
                    c0:6d:3c:c4:54:00:09:6e:f7:27:5c:fc:79:d8:2c:
                    3d:ab:99:eb:50:76:d2:70:bc:a3:c8:b5:4e:31:b1:
                    b8:2a:68:25:9f:41:b4:6d:59:b7:48:6f:cf:a9:10:
                    30:75:9c:b8:1f:8c:36:42:f7:c1:6d:b9:35:0f:e9:
                    0c:b6:90:9a:bd:25:b4:b5:e2:d7:bd:f5:71:9d:a6:
                    a5:20:de:34:81:8c:58:8d:67:b9:ee:55:3d:e7:65:
                    xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx
                    42:49:23:7c:cd:0c:81:9a:19:35:50:11:9f:44:27:
                    2e:32:8f:a0:50:fc:0e:b5:fa:f1:e9:17:47:b1:db:
                    6f:40:59:43:4d:78:e4:59:0f:c3:b1:74:2e:ee:41:
                    34:6a:9e:2c:c7:78:de:4f:c0:db:33:73:a6:c3:b2:
                    c8:25:c4:12:95:71:ea:b6:1d:bd:de:9c:a5:21:2c:
                    92:b8:f7:11:5f:46:76:3c:f7:58:fe:97:e3:b6:8b:
                    be:f6:0e:99:ff:f0:53:91:f9:c3:6e:2d:08:58:68:
                    7f:13:8b:d2:cb:6d:76:74:64:e0:c3:60:01:ea:6d:
                    32:08:2c:bf:ba:e3:f0:6f:d6:18:7b:ff:b4:1a:9c:
                    00:ca:7f:e2:a8:a1:12:32:0e:6a:f0:7a:b9:42:ac:
                    e1:b7:a7
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            X509v3 Key Usage:
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            Netscape Cert Type:
                SSL Server
            Netscape Comment:
                Katello SSL Tool Generated Certificate
            X509v3 Subject Key Identifier:
                08:4C:FF:D4:D0:D9:BA:14:63:EA:CE:EA:37:7C:BA:9C:7C:A9:7F:71
            X509v3 Authority Key Identifier:
                keyid:5B:8B:70:09:8F:C4:CA:BD:00:B8:A0:F3:22:24:4B:C8:53:8A:7C:DF
                DirName:/C=US/ST=North Carolina/L=Raleigh/O=Katello/OU=SomeOrgUnit/CN=sjprdsatapp01.cadence.com
                serial:0C:75:72:CB:32:A3:F7:8B:19:66:E2:47:20:FE:02:39:38:66:34:CB

            X509v3 Subject Alternative Name:
                DNS:sjpuppetm02.cadence.com
    Signature Algorithm: sha256WithRSAEncryption
         xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:
         1e:86:75:50:3b:ac:72:70:e9:73:e8:66:d4:45:3e:63:3c:a1:
         47:7e:54:1e:0d:f9:e5:95:a6:dc:f4:5e:8c:34:57:5b:28:83:
         9b:19:cd:9e:70:b8:1d:51:bf:3d:e5:b7:2d:61:13:2b:1e:de:
         b3:ca:ef:1a:76:39:d3:02:3a:3f:6d:ef:88:38:9f:5c:ed:d3:
         e7:c1:ae:d9:0c:a0:1c:e4:12:e0:d5:51:06:ca:4e:c8:dc:e2:
         4a:4b:72:cf:5e:88:5c:ed:bc:76:28:a1:dc:83:f1:85:47:19:
         08:ad:1a:0d:02:8c:21:e5:e2:69:5a:64:23:18:8d:68:fd:25:
         b4:4b:15:2e:32:63:39:a8:6f:de:75:df:9c:f8:b5:0b:fc:6e:
         97:95:c2:74:23:b7:11:bb:5a:53:7c:f5:6b:c5:b7:b6:b8:30:
         b7:5f:68:e5:90:76:e5:46:a0:db:71:b9:7c:e1:2a:fa:86:9c:
         b8:4f:97:da:a9:30:f7:cf:5a:26:83:f6:00:3e:90:13:11:82:
         94:82:83:6b:a8:e5:a6:77:e8:2b:61:c4:8b:93:ae:04:f8:71:
         92:85:70:ce:2c:e4:c1:64:c3:31:96:ed:cb:eb:d8:9e:fa:74:
         37:85:f0:5a:5b:53:a6:e7:81:2c:f4:54:5c:c3:e7:43:3d:66:
         e3:65:2c:c2:03:bb:e1:be:35:11:3e:3a:80:1b:d9:12:a1:cf:
         xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:xx:
         03:b5:de:51:63:25:bd:56:73:ad:d7:72:6d:35:71:78:f0:65:
         c3:5d:ff:0a:5b:a4:cf:cd:c8:87:3d:d6:34:21:a3:1e:95:cb:
         4b:ab:0f:b3:e1:3d:da:de:3f:7d:78:4e:5d:91:f9:9f:9e:82:
         3b:ba:71:de:ee:14:d5:06:40:40:e3:9b:82:46:ed:7c:06:94:
         bc:95:12:88:04:78:53:8a:19:35:e3:2e:a9:15:5e:06:86:11:
         09:86:13:36:27:87:08:d9:6a:3c:93:8f:a1:1a:62:62:fa:5d:
         a8:25:21:84:f1:b1:a9:8e:76:91:cc:be:1a:72:42:76:5f:97:
         a7:de:fb:a2:c1:8f:86:d0:49:33:a4:dc:68:a4:5d:69:d3:58:
         cb:88:41:d1:18:0c:d8:0e:d8:c7:d3:55:6c:5a:e9:20:21:5b:
         d3:83:06:6e:8c:c0:3c:6d:0d:4e:fd:76:cc:25:db:28:b0:32:
         32:99:e3:70:eb:fc:c1:54:f6:86:0c:0c:f7:84:d5:6c:48:cf:
         6e:26:45:da:c2:e1:ce:79

2023-10-09T05:38:05  [I] WEBrick::HTTPServer#start: pid=48139 port=9090
2023-10-09T05:38:05  [I] Smart proxy has launched on 2 socket(s), waiting for requests
2023-10-09T05:38:06 048c5a24 [I] Started GET /v2/features
2023-10-09T05:38:06 048c5a24 [I] Finished GET /v2/features with 200 (279.4 ms)
2023-10-09T05:38:06 a904c8fe [I] Started GET /v2/features
2023-10-09T05:38:06 a904c8fe [I] Finished GET /v2/features with 200 (27.58 ms)
2023-10-09T05:38:06 a904c8fe [I] Started GET /v2/features
2023-10-09T05:38:07 a904c8fe [I] Finished GET /v2/features with 200 (274.49 ms)
2023-10-09T05:38:55 b51ebd4c [I] Started GET /version
2023-10-09T05:38:55 b51ebd4c [I] Finished GET /version with 200 (0.84 ms)
2023-10-10T04:48:07 cc30f538 [I] Started GET /version
2023-10-10T04:48:07 cc30f538 [I] Finished GET /version with 200 (0.29 ms)

In order for foreman server to use puppet smart proxy, puppet plugin needs to be enabled on foreman server.

Enabled it using below command.
foreman-installer --enable-foreman-plugin-puppet --enable-foreman-cli-puppet --enable-puppet