Smart proxy "puppet" feature configured but not available

Problem:
I have configured puppet plugin for smart proxy, but it is not shown neither as Active nor as Failed in Foreman web-interface. However, direct request to smart proxy API says that puppet plugin is enabled:
curl -k https://foremansp.localdomain.net:8443/v2/features

“puppet”:{“http_enabled”:false,“https_enabled”:true,“settings”:{“puppet_url”:“https://puppet.localdomain.net:8140”,“use_provider”:[“puppet_proxy_puppet_api”]},“state”:“running”,“capabilities”:}

Expected outcome:
Puppet feature shown in Foreman as Active and working
Foreman and Proxy versions:
Foreman 3.6 & Proxy 3.6.1
Distribution and version:
Dockerized setup with Debian 11 (Bullseye) as base image
Other relevant data:

Hello!
I have following setup in Docker:
Container1 = Puppetserver&PuppetCA
Container2 = Foreman
Container3 = SmartProxy

I’m enabling smart proxy’s plugins one by one. So far Logs and PuppetCA plugins are configured and working fine.
Now its time to get Puppet plugin working, so I configured Puppet plugin .yml files and auth.conf for puppetserver exactly as in manual (Foreman :: Manual).

Here is smart proxy startup log (debug level):

=============================

2023-05-12T07:53:24  [W] Couldn't find settings file /usr/share/foreman-proxy/config/settings.d/facts.yml. Using default settings.
2023-05-12T07:53:24  [W] Couldn't find settings file /usr/share/foreman-proxy/config/settings.d/dns.yml. Using default settings.
2023-05-12T07:53:24  [W] Couldn't find settings file /usr/share/foreman-proxy/config/settings.d/templates.yml. Using default settings.
2023-05-12T07:53:24  [W] Couldn't find settings file /usr/share/foreman-proxy/config/settings.d/tftp.yml. Using default settings.
2023-05-12T07:53:24  [W] Couldn't find settings file /usr/share/foreman-proxy/config/settings.d/dhcp.yml. Using default settings.
2023-05-12T07:53:24  [D] 'puppetca' settings: 'enabled': true, 'puppet_version': 7.9.2, 'use_provider': ["puppetca_hostname_whitelisting", :puppetca_http_api]
2023-05-12T07:53:24  [D] 'puppetca' ports: 'http': true, 'https': true
**2023-05-12T07:53:24  [D] 'puppet' settings: 'enabled': https, 'puppet_version': 7.9.2, 'use_provider': [:puppet_proxy_puppet_api]**
2023-05-12T07:53:24  [D] 'puppet' ports: 'http': false, 'https': true
2023-05-12T07:53:24  [W] Couldn't find settings file /usr/share/foreman-proxy/config/settings.d/bmc.yml. Using default settings.
2023-05-12T07:53:24  [W] Couldn't find settings file /usr/share/foreman-proxy/config/settings.d/realm.yml. Using default settings.
2023-05-12T07:53:24  [D] 'logs' settings: 'enabled': https
2023-05-12T07:53:24  [D] 'logs' ports: 'http': false, 'https': true
2023-05-12T07:53:24  [W] Couldn't find settings file /usr/share/foreman-proxy/config/settings.d/httpboot.yml. Using default settings.
2023-05-12T07:53:24  [W] Couldn't find settings file /usr/share/foreman-proxy/config/settings.d/registration.yml. Using default settings.
2023-05-12T07:53:24  [D] Providers ['puppetca_hostname_whitelisting', 'puppetca_http_api'] are going to be configured for 'puppetca'
**2023-05-12T07:53:24  [D] Providers ['puppet_proxy_puppet_api'] are going to be configured for 'puppet'**
2023-05-12T07:53:24  [D] 'puppetca_http_api' settings: 'puppet_ssl_ca': /etc/foreman-proxy/ssl/certs/ca.pem, 'puppet_ssl_cert': /etc/foreman-proxy/ssl/certs/puppet.localdomain.net.pem, 'puppet_ssl_key': /etc/foreman-proxy/ssl/private_keys/puppet.localdomain.net.key, 'puppet_url': https://puppet.localdomain.net:8140, 'puppet_version': 7.9.2, 'use_provider': ["puppetca_hostname_whitelisting", :puppetca_http_api]
2023-05-12T07:53:24  [D] 'puppetca_hostname_whitelisting' settings: 'autosignfile': /etc/puppetlabs/puppet/autosign.conf, 'puppet_version': 7.9.2, 'use_provider': ["puppetca_hostname_whitelisting", :puppetca_http_api]
**2023-05-12T07:53:24  [D] 'puppet_proxy_puppet_api' settings: 'api_timeout': 30 (default), 'classes_retriever': apiv3, 'environments_retriever': apiv3, 'puppet_ssl_ca': /etc/foreman-proxy/ssl/certs/ca.pem, 'puppet_ssl_cert': /etc/foreman-proxy/ssl/certs/puppet.localdomain.net.pem, 'puppet_ssl_key': /etc/foreman-proxy/ssl/private_keys/puppet.localdomain.net.key, 'puppet_url': https://puppet.localdomain.net:8140, 'puppet_version': 7.9.2, 'use_provider':** [:puppet_proxy_puppet_api]
2023-05-12T07:53:24  [I] Successfully initialized 'foreman_proxy'
2023-05-12T07:53:24  [I] Successfully initialized 'puppetca_http_api'
2023-05-12T07:53:24  [I] Successfully initialized 'puppetca_hostname_whitelisting'
2023-05-12T07:53:24  [I] Successfully initialized 'puppetca'
**2023-05-12T07:53:24  [I] Successfully initialized 'puppet_proxy_puppet_api'**
**2023-05-12T07:53:24  [I] Successfully initialized 'puppet'**
2023-05-12T07:53:24  [D] Log buffer API initialized, available capacity: 2000/1000
2023-05-12T07:53:24  [I] Successfully initialized 'logs'
2023-05-12T07:53:24  [I] WEBrick 1.6.1
2023-05-12T07:53:24  [I] ruby 2.7.4 (2021-07-07) [x86_64-linux-gnu]
2023-05-12T07:53:24  [I] 
Certificate:
---removed---
2023-05-12T07:53:24  [D] Rack::Handler::WEBrick is mounted on /.
2023-05-12T07:53:24  [I] WEBrick::HTTPServer#start: pid=1 port=8443
2023-05-12T07:53:24  [I] Smart proxy has launched on 2 socket(s), waiting for requests

=============================
All seems to be OK, puppet plugin initialized successfully, but:

(Dont look at 9 warnings - these are for missing yml files, see log above)

My configs:

puppet.yml:
---
# Can be true, false, or http/https to enable just one of the protocols
:enabled: https
:puppet_version: 7.9.2

puppet_proxy_puppet_api.yml:
---
# URL of the puppet master itself for API requests.
:puppet_url: <new users cant post more than 5 links>
#
# SSL certificates used to access the puppet API
:puppet_ssl_ca: /etc/foreman-proxy/ssl/certs/ca.pem
:puppet_ssl_cert: /etc/foreman-proxy/ssl/certs/puppet.localdomain.net.pem
:puppet_ssl_key: /etc/foreman-proxy/ssl/private_keys/puppet.localdomain.net.key
#
# Smart Proxy api timeout when Puppet's environment classes api is used and classes cache is disabled
:api_timeout: 30

I also tried to explicitly set provider to puppet_proxy_puppet_api and change/remove puppet_version value in puppet.yml, without any success.

Only thing i can think of this now is that maybe smart proxy supports only puppet versions 5 and 6, as mentioned in manual section 4.3, but i believe version 7 should be supported too. If so, can anyone tell me where to look next?

Does a register happen after the plugin was configured and the service restarted? If the information is not pushed in this way from Smart Proxy to Foreman, you can also refresh the features in the Foreman which will query the Smart Proxy.

Sorry, what is register?

I’m usually restarting both Foreman and Smart proxy after any change in configs.
When i use “Refresh features” button, here is what happens on smart proxy side:

172.17.0.1 - - [12/May/2023:09:59:03 UTC] "GET /v2/features HTTP/1.0" 200 1632
- -> /v2/features
172.17.0.1 - - [12/May/2023:09:59:03 UTC] "GET /v2/features HTTP/1.0" 200 1632
- -> /v2/features
172.17.0.1 - - [12/May/2023:09:59:04 UTC] "GET /version HTTP/1.0" 200 82
- -> /version
172.17.0.1 - - [12/May/2023:09:59:04 UTC] "GET /logs/?from_timestamp=0 HTTP/1.0" 200 38548
- -> /logs/?from_timestamp=0
172.17.0.1 - - [12/May/2023:09:59:04 UTC] "GET /puppet/ca/autosign HTTP/1.0" 200 2
- -> /puppet/ca/autosign
172.17.0.1 - - [12/May/2023:09:59:04 UTC] "GET /puppet/ca HTTP/1.0" 200 361957
- -> /puppet/ca
172.17.0.1 - - [12/May/2023:09:59:04 UTC] "GET /puppet/ca HTTP/1.0" 200 361957
- -> /puppet/ca
172.17.0.1 - - [12/May/2023:09:59:04 UTC] "GET /puppet/ca HTTP/1.0" 200 361957
- -> /puppet/ca

and thats for Foreman side:

2023-05-12T09:59:03 [I|app|b98c3749] Started PUT "/smart_proxies/1-foremansp/refresh" for 172.17.0.1 at 2023-05-12 09:59:03 +0000
2023-05-12T09:59:03 [I|app|b98c3749] Processing by SmartProxiesController#refresh as HTML
2023-05-12T09:59:03 [I|app|b98c3749]   Parameters: {"authenticity_token"=>"guMvkuqnzlGu/VWc8nv0nET8VICm8H7A/l1nYM4/fhAYL0IqyIH9hysn4O53BF3AF+al+WGrUgJ2haTWerCusg==", "id"=>"1-foremansp"}
2023-05-12T09:59:03 [I|app|b98c3749] Redirected to https://foreman.localdomain.net/smart_proxies/1-foremansp
2023-05-12T09:59:03 [I|app|b98c3749] Completed 302 Found in 127ms (ActiveRecord: 20.3ms | Allocations: 13579)
2023-05-12T09:59:03 [I|app|00bf6df4] Started GET "/smart_proxies/1-foremansp" for 172.17.0.1 at 2023-05-12 09:59:03 +0000
2023-05-12T09:59:03 [I|app|00bf6df4] Processing by SmartProxiesController#show as HTML
2023-05-12T09:59:03 [I|app|00bf6df4]   Parameters: {"id"=>"1-foremansp"}
2023-05-12T09:59:03 [I|app|00bf6df4]   Rendered smart_proxies/show.html.erb within layouts/application (Duration: 45.3ms | Allocations: 12757)
2023-05-12T09:59:03 [I|app|00bf6df4]   Rendered layouts/base.html.erb (Duration: 14.5ms | Allocations: 10644)
2023-05-12T09:59:03 [I|app|00bf6df4]   Rendered layout layouts/application.html.erb (Duration: 61.0ms | Allocations: 23920)
2023-05-12T09:59:03 [I|app|00bf6df4] Completed 200 OK in 79ms (Views: 45.5ms | ActiveRecord: 22.7ms | Allocations: 27750)
2023-05-12T09:59:03 [I|app|35327988] Started GET "/notification_recipients" for 172.17.0.1 at 2023-05-12 09:59:03 +0000
2023-05-12T09:59:03 [I|app|35327988] Processing by NotificationRecipientsController#index as JSON
2023-05-12T09:59:03 [I|app|35327988] Completed 200 OK in 9ms (Views: 0.1ms | ActiveRecord: 2.6ms | Allocations: 1866)
2023-05-12T09:59:03 [I|app|fd4d7d34] Started GET "/smart_proxies/1-foremansp/ping" for 172.17.0.1 at 2023-05-12 09:59:03 +0000
2023-05-12T09:59:04 [I|app|fd4d7d34] Processing by SmartProxiesController#ping as */*
2023-05-12T09:59:04 [I|app|fd4d7d34]   Parameters: {"id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|fd4d7d34] Completed 200 OK in 43ms (Views: 0.4ms | ActiveRecord: 6.4ms | Allocations: 5465)
2023-05-12T09:59:04 [I|app|cee65783] Started GET "/smart_proxies/1-foremansp/failed_modules?smart_proxy_id=1-foremansp" for 172.17.0.1 at 2023-05-12 09:59:04 +0000
2023-05-12T09:59:04 [I|app|cee65783] Processing by SmartProxiesController#failed_modules as HTML
2023-05-12T09:59:04 [I|app|cee65783]   Parameters: {"smart_proxy_id"=>"1-foremansp", "id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|cee65783] Completed 200 OK in 48ms (Views: 0.7ms | ActiveRecord: 8.7ms | Allocations: 7860)
2023-05-12T09:59:04 [I|app|029cf05d] Started GET "/smart_proxies/1-foremansp/autosign/1-foremansp/counts" for 172.17.0.1 at 2023-05-12 09:59:04 +0000
2023-05-12T09:59:04 [I|app|029cf05d] Processing by AutosignController#counts as HTML
2023-05-12T09:59:04 [I|app|029cf05d]   Parameters: {"smart_proxy_id"=>"1-foremansp", "id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|029cf05d] Completed 200 OK in 43ms (Views: 0.9ms | ActiveRecord: 6.0ms | Allocations: 5354)
2023-05-12T09:59:04 [I|app|2dcfc313] Started GET "/smart_proxies/1-foremansp/puppetca" for 172.17.0.1 at 2023-05-12 09:59:04 +0000
2023-05-12T09:59:04 [I|app|2dcfc313] Processing by PuppetcaController#index as HTML
2023-05-12T09:59:04 [I|app|2dcfc313]   Parameters: {"smart_proxy_id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|c448f131] Started GET "/smart_proxies/1-foremansp/autosign" for 172.17.0.1 at 2023-05-12 09:59:04 +0000
2023-05-12T09:59:04 [I|app|1bd13b2c] Started GET "/smart_proxies/1-foremansp/puppetca/1-foremansp/counts" for 172.17.0.1 at 2023-05-12 09:59:04 +0000
2023-05-12T09:59:04 [I|app|c9cca787] Started GET "/smart_proxies/1-foremansp/errors_card?smart_proxy_id=1-foremansp" for 172.17.0.1 at 2023-05-12 09:59:04 +0000
2023-05-12T09:59:04 [I|app|c448f131] Processing by AutosignController#index as HTML
2023-05-12T09:59:04 [I|app|c448f131]   Parameters: {"smart_proxy_id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|c9cca787] Processing by SmartProxiesController#errors_card as HTML
2023-05-12T09:59:04 [I|app|c9cca787]   Parameters: {"smart_proxy_id"=>"1-foremansp", "id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|54b454ed] Started GET "/smart_proxies/1-foremansp/autosign/new" for 172.17.0.1 at 2023-05-12 09:59:04 +0000
2023-05-12T09:59:04 [I|app|7b5cb59c] Started GET "/smart_proxies/1-foremansp/puppetca/1-foremansp/expiry" for 172.17.0.1 at 2023-05-12 09:59:04 +0000
2023-05-12T09:59:04 [I|app|1bd13b2c] Processing by PuppetcaController#counts as HTML
2023-05-12T09:59:04 [I|app|1bd13b2c]   Parameters: {"smart_proxy_id"=>"1-foremansp", "id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|c9cca787] Completed 200 OK in 27ms (Views: 2.6ms | ActiveRecord: 9.9ms | Allocations: 10667)
2023-05-12T09:59:04 [I|app|73b596ac] Started GET "/smart_proxies/1-foremansp/modules_card?smart_proxy_id=1-foremansp" for 172.17.0.1 at 2023-05-12 09:59:04 +0000
2023-05-12T09:59:04 [I|app|c448f131] Completed 200 OK in 39ms (Views: 1.6ms | ActiveRecord: 18.0ms | Allocations: 9551)
2023-05-12T09:59:04 [I|app|73b596ac] Processing by SmartProxiesController#modules_card as HTML
2023-05-12T09:59:04 [I|app|73b596ac]   Parameters: {"smart_proxy_id"=>"1-foremansp", "id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|54b454ed] Processing by AutosignController#new as HTML
2023-05-12T09:59:04 [I|app|54b454ed]   Parameters: {"smart_proxy_id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|a9b80e90] Started GET "/smart_proxies/1-foremansp/log_pane?smart_proxy_id=1-foremansp" for 172.17.0.1 at 2023-05-12 09:59:04 +0000
2023-05-12T09:59:04 [I|app|7b5cb59c] Processing by PuppetcaController#expiry as HTML
2023-05-12T09:59:04 [I|app|7b5cb59c]   Parameters: {"smart_proxy_id"=>"1-foremansp", "id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|73b596ac] Completed 200 OK in 51ms (Views: 4.7ms | ActiveRecord: 30.4ms | Allocations: 16730)
2023-05-12T09:59:04 [I|app|54b454ed] Completed 200 OK in 60ms (Views: 5.3ms | ActiveRecord: 29.2ms | Allocations: 15766)
2023-05-12T09:59:04 [I|app|a9b80e90] Processing by SmartProxiesController#log_pane as HTML
2023-05-12T09:59:04 [I|app|a9b80e90]   Parameters: {"smart_proxy_id"=>"1-foremansp", "id"=>"1-foremansp"}
2023-05-12T09:59:04 [I|app|a9b80e90] Completed 200 OK in 235ms (Views: 200.9ms | ActiveRecord: 11.7ms | Allocations: 272114)

Both seems OK. I’m sure Foreman gets puppet plugin settings from /v2/features request to api, but somewhere fails to interpret it.

I was referring to the API call you will see using the installer or puppet module that registers the Smart proxy to Foreman which sends the list of features from the Smart Proxy to Foreman.

Yes, I also do not see an error. I am not sure if Foreman requires the Puppet plugin already at this stage, but just to ensure: Is the Foreman Puppet plugin installed?

I installed Foreman from source, not sure if installer script ran at all…

This one?
I think no, at least i did not installed it myself. I just followed manual and it seems like i missed something about plugins…
How can i quickly check if it is already installed?

Yes, Puppet was extracted to live in a separate plugin with version 3.0, so if you want all features of the Puppet integration you will need this one.

Installed plugins are shown on the Administer > About page.