Some fact values will not be updated

Support
1

Problem:
If I add a new host, not all facts will be update like ip, os, Mac, model etc. First run fails with the following error:

AGENT:

Debug: Creating new connection for https://puppet:8140
Debug: Starting connection for https://puppet:8140
Debug: Using TLSv1.2 with cipher DHE-RSA-AES128-GCM-SHA256
Debug: HTTP POST https://puppet:8140/puppet/v3/catalog/satisfactory?environment=production returned 500 Server Error
Debug: Caching connection for https://puppet:8140
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node satisfactory: Failed to find satisfactory via exec: Execution of '/etc/puppetlabs/puppet/node.rb satisfactory' returned 1:
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Debug: Resolving service 'report' using Puppet::HTTP::Resolver::Settings
Debug: Using cached connection for https://puppet:8140
Debug: Caching connection for https://puppet:8140
Debug: Resolved service 'report' to https://puppet:8140/puppet/v3
Debug: Puppet::Network::Format[msgpack]: feature msgpack is missing
Debug: Puppet::Network::Format[rich_data_msgpack]: feature msgpack is missing
Debug: report supports formats: json pson yaml
Debug: Using cached connection for https://puppet:8140
Debug: HTTP PUT https://puppet:8140/puppet/v3/report/satisfactory?environment=production returned 200 OK
Debug: Caching connection for https://puppet:8140
Debug: Closing connection for https://puppet:8140

SERVER:

2023-04-20T11:17:07.737+02:00 ERROR [qtp901154776-754] [p.s.c.certificate-authority-core] satisfactory already has a requested certificate; ignoring certificate request
2023-04-20T11:17:08.857+02:00 INFO  [qtp901154776-754] [p.p.certificate-authority] Signed certificate request for satisfactory
2023-04-20T11:17:08.915+02:00 ERROR [qtp901154776-754] [p.s.c.certificate-authority-core] satisfactory already has a signed certificate; ignoring certificate request
2023-04-20T11:17:09.639+02:00 WARN  [qtp901154776-685] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: /etc/puppetlabs/puppet/node.rb:447: warning: constant ::TimeoutError is deprecated
Serving cached ENC: Error retrieving node satisfactory: Net::HTTPNotFound
Check Foreman's /var/log/foreman/production.log for more information.
Unable to read from Cache file: No such file or directory @ rb_sysopen - /opt/puppetlabs/server/data/puppetserver/yaml/foreman/satisfactory.yaml

2023-04-20T11:17:09.648+02:00 ERROR [qtp901154776-685] [puppetserver] Puppet Server Error: Failed to find satisfactory via exec: Execution of '/etc/puppetlabs/puppet/node.rb satisfactory' returned 1:
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:50:in `execute'
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:14:in `block in initialize_execution_stub'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:211:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:35:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/exec.rb:19:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:17:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:120:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:52:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/server/v3.rb:17:in `block in wrap'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1865:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:69:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:65:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:92:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:84:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:69:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:68:in `handleRequest'
2023-04-20T11:17:32.804+02:00 INFO  [qtp901154776-664] [puppetserver] Puppet 'replace_facts' command for satisfactory submitted to PuppetDB with UUID ae8cd92d-17e3-4c4a-bac7-ec09f2e19c6f
2023-04-20T11:17:33.358+02:00 WARN  [qtp901154776-664] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: /etc/puppetlabs/puppet/node.rb:447: warning: constant ::TimeoutError is deprecated
Serving cached ENC: Error retrieving node satisfactory: Net::HTTPNotFound
Check Foreman's /var/log/foreman/production.log for more information.
Unable to read from Cache file: No such file or directory @ rb_sysopen - /opt/puppetlabs/server/data/puppetserver/yaml/foreman/satisfactory.yaml

2023-04-20T11:17:33.503+02:00 ERROR [qtp901154776-664] [puppetserver] Puppet Failed when searching for node satisfactory: Failed to find satisfactory via exec: Execution of '/etc/puppetlabs/puppet/node.rb satisfactory' returned 1:
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:50:in `execute'
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:14:in `block in initialize_execution_stub'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:211:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:35:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/exec.rb:19:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:17:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:368:in `block in find_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:365:in `find_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:407:in `node_from_request'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:52:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:120:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:52:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/server/v3.rb:17:in `block in wrap'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1865:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:69:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:65:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:92:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:84:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:69:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:68:in `handleRequest'
2023-04-20T11:17:33.579+02:00 ERROR [qtp901154776-664] [puppetserver] Puppet Server Error: Failed when searching for node satisfactory: Failed to find satisfactory via exec: Execution of '/etc/puppetlabs/puppet/node.rb satisfactory' returned 1:
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:50:in `execute'
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:14:in `block in initialize_execution_stub'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:211:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:35:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/exec.rb:19:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:17:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:368:in `block in find_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:365:in `find_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:407:in `node_from_request'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:52:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:120:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:52:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/server/v3.rb:17:in `block in wrap'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1865:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:69:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:65:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:92:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:84:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:69:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:68:in `handleRequest'
2023-04-20T11:17:37.082+02:00 INFO  [qtp901154776-784] [puppetserver] Puppet 'store_report' command for satisfactory submitted to PuppetDB with UUID 0b59c823-6e4a-4b05-9243-f6807588f6f1

cat: /opt/puppetlabs/server/data/puppetserver/yaml/foreman/satisfactory.yaml: No such file or directory

if i execute /etc/puppetlabs/puppet/node.rb satisfactory i fot content, but without a lot of values

---
classes: []
parameters:
  puppetmaster: ''
  foreman_hostname: satisfactory
  foreman_fqdn: satisfactory
  root_pw:
  foreman_subnets: []
  foreman_interfaces:
  - ip:
    ip6:
    mac:
    name: satisfactory
    attrs: {}
    virtual: false
    link: true
    identifier:
    managed: true
    primary: true
    provision: true
    subnet:
    subnet6:
    tag:
    attached_to:
    type: Interface
  host_packages: ''
  host_registration_insights: false
  host_registration_remote_execution: true

and now the file /opt/puppetlabs/server/data/puppetserver/yaml/foreman/satisfactory.yaml exists:

---
classes: []
parameters:
  puppetmaster: ''
  foreman_hostname: satisfactory
  foreman_fqdn: satisfactory
  root_pw:
  foreman_subnets: []
  foreman_interfaces:
  - ip:
    ip6:
    mac:
    name: satisfactory
    attrs: {}
    virtual: false
    link: true
    identifier:
    managed: true
    primary: true
    provision: true
    subnet:
    subnet6:
    tag:
    attached_to:
    type: Interface
  host_packages: ''
  host_registration_insights: false
  host_registration_remote_execution: true

Same, if i run the agent a second time, no errors are seeing, but the values are empty and will not be updated or filled with params.

And this happens after upgrading Foreman.

puppet facts are give all informations about the host on the host, but they will noch be updated to the Foreman host. Reports are given, but facts not.

See attached Screenshots

Hope you have a easy solution for this issue. BR. Torsten

Foreman and Proxy versions:

3.6.1

Foreman and Proxy plugin versions:

foreman_fog_proxmox 0.14.0
foreman_puppet 5.1.0
puppetdb_foreman 6.0.1

Distribution and version:

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
Puppet: 7.24.0
2

config_reports
config_reports3640×722 107 KB

fact_reports
fact_reports3532×418 33.9 KB

host_status
host_status3534×522 57.5 KB

3

From what I can see, the problem is most likely that your hosts are reporting their shortname (without domain) as their Puppet hostname, but Foreman knows them by fqdn. This is most likely a configuration issue on your hosts’ side. You should check your agent’s puppet.conf (and where-ever you are deploying the initial puppet.conf) for this.
I don’t really understand why the node.rb is still putting a near-empty yaml file in the cache, but that is more like a side-effect of the actual ENC queries not working.

4

In the logs, I removed the full fqdn. What I can see is that nothing will upload. I will add the fqdn in the puppet.conf as well and see what happened.

5

What do you see in production.log when a host performs a puppetrun?
There should be entries like these:
ENC query:

2023-04-25T01:33:23 [I|app|20d593df] Started GET "/node/<fqdn>?format=yml" for <puppetserver-ip> at 2023-04-25 01:33:23 +0200
2023-04-25T01:33:23 [I|app|20d593df] Processing by HostsController#externalNodes as YML
2023-04-25T01:33:23 [I|app|20d593df]   Parameters: {"name"=>"<fqdn>"}
2023-04-25T01:33:23 [I|app|20d593df]   Rendering text template
2023-04-25T01:33:23 [I|app|20d593df]   Rendered text template (Duration: 0.0ms | Allocations: 1)
2023-04-25T01:33:23 [I|app|20d593df] Completed 200 OK in 370ms (Views: 0.4ms | ActiveRecord: 77.7ms | Allocations: 257987)

Fact upload:

2023-04-25T01:33:22 [I|app|ddba1bec] Started POST "/api/hosts/facts" for <puppetserver-IP> at 2023-04-25 01:33:22 +0200
2023-04-25T01:33:22 [I|app|ddba1bec] Processing by Api::V2::HostsController#facts as JSON
2023-04-25T01:33:22 [I|app|ddba1bec]   Parameters: {"facts"=>"[FILTERED]", "name"=>"<fqdn>", "certname"=>"<fqdn>", "apiv"=>"v2", "host"=>{"certname"=>"<fqdn>", "name"=>"<fqdn>"}}
2023-04-25T01:33:23 [I|app|ddba1bec] Import facts for '<fqdn>' completed. Added: 11, Updated: 36, Deleted 11 facts
2023-04-25T01:33:23 [I|app|ddba1bec] Completed 201 Created in 440ms (Views: 8.5ms | ActiveRecord: 90.3ms | Allocations: 113048)

Report upload:

2023-04-25T01:33:23 [I|app|e5191e8a] Started POST "/api/config_reports" for <puppetserver-ip> at 2023-04-25 01:33:23 +0200
2023-04-25T01:33:23 [I|app|e5191e8a] Processing by Api::V2::ConfigReportsController#create as JSON
2023-04-25T01:33:23 [I|app|e5191e8a]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2023-04-25T01:33:23 [I|app|e5191e8a] Scanning report with: Foreman::PuppetReportScanner
2023-04-25T01:33:23 [I|app|e5191e8a] Imported report for <fqdn> in 33.3 ms, status refreshed in 60.1 ms
2023-04-25T01:33:23 [I|app|e5191e8a]   Rendering api/v2/config_reports/create.json.rabl
2023-04-25T01:33:23 [I|app|e5191e8a]   Rendered api/v2/config_reports/create.json.rabl (Duration: 8.3ms | Allocations: 7003)
2023-04-25T01:33:23 [I|app|e5191e8a] Completed 201 Created in 118ms (Views: 10.1ms | ActiveRecord: 30.0ms | Allocations: 38270)
6

I Installed a new Testsystem calling puppet test.mydomain.ltd. Here you can see (hope I got all log messages) what happened in the first puppet run step:

puppetserver log:

2023-04-26T09:13:33.771+02:00 INFO  [qtp901154776-28021] [p.p.certificate-authority] Signed certificate request for puppettest.mydomain.ltd
2023-04-26T09:13:34.848+02:00 ERROR [qtp901154776-28042] [p.s.c.certificate-authority-core] puppettest.mydomain.ltd already has a signed certificate; ignoring certificate request
2023-04-26T09:13:35.560+02:00 WARN  [qtp901154776-27940] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: /etc/puppetlabs/puppet/node.rb:447: warning: constant ::TimeoutError is deprecated
Serving cached ENC: Error retrieving node puppettest.mydomain.ltd: Net::HTTPNotFound
Check Foreman's /var/log/foreman/production.log for more information.
Unable to read from Cache file: No such file or directory @ rb_sysopen - /opt/puppetlabs/server/data/puppetserver/yaml/foreman/puppettest.mydomain.ltd.yaml

2023-04-26T09:13:35.700+02:00 ERROR [qtp901154776-27940] [puppetserver] Puppet Server Error: Failed to find puppettest.mydomain.ltd via exec: Execution of '/etc/puppetlabs/puppet/node.rb puppettest.mydomain.ltd' returned 1:
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:50:in `execute'
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:14:in `block in initialize_execution_stub'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:211:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:35:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/exec.rb:19:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:17:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:120:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:52:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/server/v3.rb:17:in `block in wrap'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1865:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:69:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:65:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:92:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:84:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:69:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:68:in `handleRequest'

2023-04-26T09:13:56.976+02:00 INFO  [qtp901154776-27720] [puppetserver] Puppet 'replace_facts' command for puppettest.mydomain.ltd submitted to PuppetDB with UUID b5471eda-75ad-48d5-a255-14407eac8cb4
2023-04-26T09:13:57.466+02:00 WARN  [qtp901154776-27720] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: /etc/puppetlabs/puppet/node.rb:447: warning: constant ::TimeoutError is deprecated
Serving cached ENC: Error retrieving node puppettest.mydomain.ltd: Net::HTTPNotFound
Check Foreman's /var/log/foreman/production.log for more information.
Unable to read from Cache file: No such file or directory @ rb_sysopen - /opt/puppetlabs/server/data/puppetserver/yaml/foreman/puppettest.mydomain.ltd.yaml

2023-04-26T09:13:57.653+02:00 ERROR [qtp901154776-27720] [puppetserver] Puppet Failed when searching for node puppettest.mydomain.ltd: Failed to find puppettest.mydomain.ltd via exec: Execution of '/etc/puppetlabs/puppet/node.rb puppettest.mydomain.ltd' returned 1:
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:50:in `execute'
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:14:in `block in initialize_execution_stub'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:211:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:35:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/exec.rb:19:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:17:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:368:in `block in find_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:365:in `find_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:407:in `node_from_request'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:52:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:120:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:52:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/server/v3.rb:17:in `block in wrap'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1865:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:69:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:65:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:92:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:84:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:69:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:68:in `handleRequest'
2023-04-26T09:13:57.677+02:00 ERROR [qtp901154776-27720] [puppetserver] Puppet Server Error: Failed when searching for node puppettest.mydomain.ltd: Failed to find puppettest.mydomain.ltd via exec: Execution of '/etc/puppetlabs/puppet/node.rb puppettest.mydomain.ltd' returned 1:
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:50:in `execute'
uri:classloader:/puppetserver-lib/puppet/server/execution.rb:14:in `block in initialize_execution_stub'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/execution.rb:211:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:35:in `execute'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/exec.rb:19:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/node/exec.rb:17:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:368:in `block in find_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:365:in `find_node'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:407:in `node_from_request'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/catalog/compiler.rb:52:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:223:in `find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:120:in `do_find'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `block in call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:52:in `call'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/server/v3.rb:17:in `block in wrap'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'
org/jruby/RubyArray.java:1865:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:69:in `block in with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:65:in `with_request_profiling'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:92:in `respond_to_errors'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:84:in `process'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:69:in `block in handleRequest'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:289:in `override'
uri:classloader:/puppetserver-lib/puppet/server/master.rb:68:in `handleRequest'

foreman log:

2023-04-26T09:13:33 [I|app|1d6c9e24] Started PUT "/smart_proxies/1-<PUPPETSERVER>/puppetca/puppettest.mydomain.ltd" for <IP> at 2023-04-26 09:13:33 +0200
2023-04-26T09:13:33 [I|app|1d6c9e24] Processing by PuppetcaController#update as HTML
2023-04-26T09:13:33 [I|app|1d6c9e24]   Parameters: {"authenticity_token"=>"<TOKEN>", "smart_proxy_id"=>"1-<PUPPETSERVER>", "id"=>"puppettest.mydomain.ltd"}
2023-04-26T09:13:33 [I|app|1d6c9e24] Redirected to https://<PUPPETSERVER>:8444/smart_proxies/1-<PUPPETSERVER>#certificates
2023-04-26T09:13:33 [I|app|1d6c9e24] Completed 302 Found in 359ms (ActiveRecord: 7.6ms | Allocations: 4269)
2023-04-26T09:13:35 [I|app|156889df] Started GET "/node/puppettest.mydomain.ltd?format=yml" for 172.30.0.25 at 2023-04-26 09:13:35 +0200
2023-04-26T09:13:35 [I|app|156889df] Processing by HostsController#externalNodes as YML
2023-04-26T09:13:35 [I|app|156889df]   Parameters: {"name"=>"puppettest.mydomain.ltd"}
2023-04-26T09:13:35 [I|app|156889df] Completed 404 Not Found in 31ms (ActiveRecord: 12.2ms | Allocations: 2627)
2023-04-26T09:13:34 [I|app|74640cc5] Started GET "/notification_recipients" for <IP> at 2023-04-26 09:13:34 +0200
2023-04-26T09:13:34 [I|app|74640cc5] Processing by NotificationRecipientsController#index as JSON
2023-04-26T09:13:34 [W|app|74640cc5] Action failed
2023-04-26T09:13:34 [I|app|74640cc5] Backtrace for 'Action failed' error (TypeError): incompatible marshal file format (can't be read)
 74640cc5 |   format version 4.8 required; 123.34 given
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/core_ext/marshal.rb:8:in `load'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/core_ext/marshal.rb:8:in `load'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:598:in `deserialize_entry'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:459:in `deserialize_entry'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:349:in `block in read_entry'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:347:in `read_entry'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:136:in `block in read_entry'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:78:in `block in fetch_entry'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:78:in `fetch'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:78:in `fetch_entry'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:134:in `read_entry'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:367:in `block in read'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:726:in `block in instrument'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:205:in `instrument'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:726:in `instrument'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:366:in `read'
 74640cc5 | /usr/share/foreman/app/services/ui_notifications/cache_handler.rb:10:in `payload'
 74640cc5 | /usr/share/foreman/app/controllers/notification_recipients_controller.rb:7:in `index'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/abstract_controller/base.rb:228:in `process_action'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 74640cc5 | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 74640cc5 | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 74640cc5 | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/audited-5.3.2/lib/audited/sweeper.rb:16:in `around'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/audited-5.3.2/lib/audited/sweeper.rb:16:in `around'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:137:in `run_callbacks'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `block in instrument'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `instrument'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.3/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/abstract_controller/base.rb:165:in `process'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:39:in `process'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal.rb:190:in `dispatch'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal.rb:254:in `dispatch'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:32:in `each'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:32:in `serve'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:842:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/apipie-rails-0.8.2/lib/apipie/static_dispatcher.rb:68:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/apipie-rails-0.8.2/lib/apipie/extractor/recorder.rb:137:in `call'
 74640cc5 | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 74640cc5 | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/apipie-rails-0.8.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/tempfile_reaper.rb:15:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/etag.rb:27:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/conditional_get.rb:27:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/head.rb:12:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 74640cc5 | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:266:in `context'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:260:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:37:in `call_app'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:28:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 74640cc5 | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/method_override.rb:24:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/runtime.rb:22:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/sendfile.rb:110:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/engine.rb:539:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `public_send'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `method_missing'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/urlmap.rb:74:in `block in call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/urlmap.rb:58:in `each'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/urlmap.rb:58:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/configuration.rb:252:in `call'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/request.rb:77:in `block in handle_request'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/request.rb:76:in `handle_request'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/server.rb:443:in `process_client'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 74640cc5 | /usr/share/foreman/vendor/ruby/2.7.0/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-04-26T09:13:34 [I|app|74640cc5]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (Duration: 0.8ms | Allocations: 104)
2023-04-26T09:13:34 [I|app|74640cc5]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 1.4ms | Allocations: 192)
2023-04-26T09:13:34 [I|app|74640cc5] Completed 500 Internal Server Error in 18ms (Views: 3.5ms | ActiveRecord: 1.7ms | Allocations: 2129)
2023-04-26T09:13:57 [I|app|879c4d5a] Started POST "/api/config_reports" for 172.30.0.25 at 2023-04-26 09:13:57 +0200
2023-04-26T09:13:57 [I|app|879c4d5a] Processing by Api::V2::ConfigReportsController#create as JSON
2023-04-26T09:13:57 [I|app|879c4d5a]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on mac
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on ip
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on type Nic::Managed
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on name puppettest.mydomain.ltd
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on host_id 21
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on subnet_id
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on domain_id
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on attrs {}
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on provider
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on username
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on password [redacted]
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on virtual false
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on link true
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on identifier
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on tag
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on attached_to
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on managed true
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on mode balance-rr
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on attached_devices
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on bond_options
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on primary true
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on provision true
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on compute_attributes {}
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on ip6
2023-04-26T09:13:57 [I|aud|879c4d5a] Nic::Managed (22) create event on subnet6_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on name puppettest.mydomain.ltd
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on root_pass
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on architecture_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on operatingsystem_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on ptable_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on medium_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on build false
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on comment
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on disk
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on installed_at
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on model_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on hostgroup_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on owner_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on owner_type
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on enabled true
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on puppet_ca_proxy_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on managed false
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on use_image
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on image_file
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on uuid
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on compute_resource_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on puppet_proxy_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on certname
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on image_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on organization_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on location_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on otp
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on realm_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on compute_profile_id
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on provision_method
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on grub_pass
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on pxe_loader
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on initiated_at
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on build_errors
2023-04-26T09:13:58 [I|aud|879c4d5a] Host::Base (21) create event on creator_id 3
2023-04-26T09:13:58 [I|app|879c4d5a] Scanning report with: Foreman::PuppetReportScanner
2023-04-26T09:14:00 [I|app|879c4d5a] Imported report for puppettest.mydomain.ltd in 2498.4 ms, status refreshed in 26.8 ms
2023-04-26T09:14:00 [I|app|879c4d5a]   Rendered api/v2/config_reports/create.json.rabl (Duration: 5.9ms | Allocations: 767)
2023-04-26T09:14:00 [I|app|879c4d5a] Completed 201 Created in 2582ms (Views: 8.9ms | ActiveRecord: 1471.8ms | Allocations: 203326)

puppet agent log:

Info: Downloaded certificate for puppettest.mydomain.ltd from https://<PUPPETSERVER>:8140/puppet-ca/v1
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Failed to find puppettest.mydomain.ltd via exec: Execution of '/etc/puppetlabs/puppet/node.rb puppettest.mydomain.ltd' returned 1:
Info: Using environment 'production'
....
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node puppettest.mydomain.ltd: Failed to find puppettest.mydomain.ltd via exec: Execution of '/etc/puppetlabs/puppet/node.rb puppettest.mydomain.ltd' returned 1:
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Second puppet agent run:

foreman logging:

2023-04-26T09:33:39.336+02:00 INFO  [qtp901154776-28021] [puppetserver] Puppet 'replace_facts' command for puppettest.mydomain.ltd submitted to PuppetDB with UUID 23efedc1-6af1-435a-8052-8aefd417015e
2023-04-26T09:33:40.016+02:00 INFO  [qtp901154776-28021] [puppetserver] Puppet Compiled catalog for puppettest.mydomain.ltd in environment production in 0.08 seconds
2023-04-26T09:33:40.017+02:00 INFO  [qtp901154776-28021] [puppetserver] Puppet Caching catalog for puppettest.mydomain.ltd
2023-04-26T09:33:40.049+02:00 INFO  [qtp901154776-28021] [puppetserver] Puppet 'replace_catalog' command for puppettest.mydomain.ltd submitted to PuppetDB with UUID 13e7ada0-ad28-45f7-bfb2-d8006c931c72
2023-04-26T09:33:40.761+02:00 INFO  [qtp901154776-27720] [puppetserver] Puppet 'store_report' command for puppettest.mydomain.ltd submitted to PuppetDB with UUID c306b976-0b32-4069-9ad7-4afd28db2742

puppet server log:

2023-04-26T09:33:39 [I|app|c1091fda] Started GET "/node/puppettest.mydomain.ltd?format=yml" for 172.30.0.25 at 2023-04-26 09:33:39 +0200
2023-04-26T09:33:39 [I|app|c1091fda] Processing by HostsController#externalNodes as YML
2023-04-26T09:33:39 [I|app|c1091fda]   Parameters: {"name"=>"puppettest.mydomain.ltd"}
2023-04-26T09:33:39 [I|app|c1091fda]   Rendered text template (Duration: 0.1ms | Allocations: 1)
2023-04-26T09:33:39 [I|app|c1091fda] Completed 200 OK in 120ms (Views: 3.3ms | ActiveRecord: 38.1ms | Allocations: 7856)
2023-04-26T09:33:40 [I|app|3f21a546] Started POST "/api/config_reports" for 172.30.0.25 at 2023-04-26 09:33:40 +0200
2023-04-26T09:33:40 [I|app|3f21a546] Processing by Api::V2::ConfigReportsController#create as JSON
2023-04-26T09:33:40 [I|app|3f21a546]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2023-04-26T09:33:40 [I|app|3f21a546] Scanning report with: Foreman::PuppetReportScanner
2023-04-26T09:33:40 [I|app|3f21a546] Imported report for puppettest.mydomain.ltd in 141.5 ms, status refreshed in 42.3 ms
2023-04-26T09:33:40 [I|app|3f21a546]   Rendered api/v2/config_reports/create.json.rabl (Duration: 7.0ms | Allocations: 809)
2023-04-26T09:33:40 [I|app|3f21a546] Completed 201 Created in 230ms (Views: 11.5ms | ActiveRecord: 101.7ms | Allocations: 18118)

puppet agent log:

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Info: Caching catalog for puppettest.mydomain.ltd
Info: Applying configuration version '1682494419'
Info: Creating state file /opt/puppetlabs/puppet/cache/state/state.yaml

but facts are empty

facts
facts1870×282 16.1 KB

any idea?

7

Concerning the missing facts:
Could you check in /etc/puppetlabs/puppet/foreman.yaml if the :facts: value is set to true? My only guess to why facts are not uploaded at all (at least there’s no request for that in the Foreman log) would be that might be set to false for some reason.
What still confuses me more, though, is that your external node qualifier does not seem to work (maybe this is also the reason facts are not uploaded, idk). Can you check what https://yourforman.yourdomain.tld//node/puppettest.mydomain.ltd?format=yml returns? In production.log i can see it gives HTTP 200, which indicates it should be working, but puppetserver seems to dislike the output.

8 
:facts: true

Result of the requested link:

---
classes: []
parameters:
  puppetmaster: ''
  foreman_hostname: puppettest.mydomain.ltd
  foreman_fqdn: puppettest.mydomain.ltd
  root_pw: 
  foreman_subnets: []
  foreman_interfaces:
  - ip: 
    ip6: 
    mac: 
    name: puppettest.mydomain.ltd
    attrs: {}
    virtual: false
    link: true
    identifier: 
    managed: true
    primary: true
    provision: true
    subnet: 
    subnet6: 
    tag: 
    attached_to: 
    type: Interface
  host_packages: ''
  host_registration_insights: false
  host_registration_remote_execution: true
9

This will come in the foreman.log after puppet agent -t again:

2023-04-26T12:38:52 [I|app|401f63bd] Started GET "/notification_recipients" for <IP> at 2023-04-26 12:38:52 +0200
2023-04-26T12:38:52 [I|app|401f63bd] Processing by NotificationRecipientsController#index as JSON
2023-04-26T12:38:52 [W|app|401f63bd] Action failed
2023-04-26T12:38:52 [I|app|401f63bd] Backtrace for 'Action failed' error (TypeError): incompatible marshal file format (can't be read)
 401f63bd | 	format version 4.8 required; 123.34 given
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/core_ext/marshal.rb:8:in `load'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/core_ext/marshal.rb:8:in `load'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:598:in `deserialize_entry'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:459:in `deserialize_entry'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:349:in `block in read_entry'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:478:in `failsafe'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/redis_cache_store.rb:347:in `read_entry'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:136:in `block in read_entry'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:78:in `block in fetch_entry'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:78:in `fetch'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:78:in `fetch_entry'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache.rb:134:in `read_entry'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:367:in `block in read'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:726:in `block in instrument'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:205:in `instrument'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:726:in `instrument'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache.rb:366:in `read'
 401f63bd | /usr/share/foreman/app/services/ui_notifications/cache_handler.rb:10:in `payload'
 401f63bd | /usr/share/foreman/app/controllers/notification_recipients_controller.rb:7:in `index'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/basic_implicit_render.rb:6:in `send_action'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/abstract_controller/base.rb:228:in `process_action'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/rendering.rb:30:in `process_action'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/abstract_controller/callbacks.rb:42:in `block in process_action'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:117:in `block in run_callbacks'
 401f63bd | /usr/share/foreman/app/controllers/concerns/foreman/controller/timezone.rb:10:in `set_timezone'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 401f63bd | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 401f63bd | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/audited-5.3.2/lib/audited/sweeper.rb:16:in `around'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/audited-5.3.2/lib/audited/sweeper.rb:16:in `around'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:126:in `block in run_callbacks'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:137:in `run_callbacks'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/abstract_controller/callbacks.rb:41:in `process_action'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/rescue.rb:22:in `process_action'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:34:in `block in process_action'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `block in instrument'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/notifications.rb:203:in `instrument'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/instrumentation.rb:33:in `process_action'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal/params_wrapper.rb:249:in `process_action'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activerecord-6.1.7.3/lib/active_record/railties/controller_runtime.rb:27:in `process_action'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/abstract_controller/base.rb:165:in `process'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionview-6.1.7.3/lib/action_view/rendering.rb:39:in `process'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal.rb:190:in `dispatch'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_controller/metal.rb:254:in `dispatch'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:50:in `dispatch'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:33:in `serve'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:50:in `block in serve'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:32:in `each'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/journey/router.rb:32:in `serve'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/routing/route_set.rb:842:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/apipie-dsl-2.5.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/apipie-rails-0.8.2/lib/apipie/static_dispatcher.rb:68:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/apipie-rails-0.8.2/lib/apipie/extractor/recorder.rb:137:in `call'
 401f63bd | /usr/share/foreman/lib/foreman/middleware/libvirt_connection_cleaner.rb:9:in `call'
 401f63bd | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/apipie-rails-0.8.2/lib/apipie/middleware/checksum_in_headers.rb:27:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/tempfile_reaper.rb:15:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/etag.rb:27:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/conditional_get.rb:27:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/head.rb:12:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/http/permissions_policy.rb:22:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/http/content_security_policy.rb:19:in `call'
 401f63bd | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:266:in `context'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/session/abstract/id.rb:260:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/cookies.rb:697:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/callbacks.rb:98:in `run_callbacks'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/callbacks.rb:26:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/debug_exceptions.rb:29:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:37:in `call_app'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/rack/logger.rb:28:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/sprockets-rails-3.4.2/lib/sprockets/rails/quiet_assets.rb:13:in `call'
 401f63bd | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/remote_ip.rb:81:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/request_id.rb:26:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/method_override.rb:24:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/runtime.rb:22:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/activesupport-6.1.7.3/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/executor.rb:14:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/sendfile.rb:110:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/ssl.rb:77:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/actionpack-6.1.7.3/lib/action_dispatch/middleware/host_authorization.rb:142:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/secure_headers-6.5.0/lib/secure_headers/middleware.rb:11:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/engine.rb:539:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `public_send'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/railties-6.1.7.3/lib/rails/railtie.rb:207:in `method_missing'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/urlmap.rb:74:in `block in call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/urlmap.rb:58:in `each'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/urlmap.rb:58:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/configuration.rb:252:in `call'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/request.rb:77:in `block in handle_request'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/request.rb:76:in `handle_request'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/server.rb:443:in `process_client'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/puma-5.6.5/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
 401f63bd | /usr/share/foreman/vendor/ruby/2.7.0/gems/logging-2.3.1/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2023-04-26T12:38:52 [I|app|401f63bd]   Rendered api/v2/errors/standard_error.json.rabl within api/v2/layouts/error_layout (Duration: 1.8ms | Allocations: 104)
2023-04-26T12:38:52 [I|app|401f63bd]   Rendered layout api/v2/layouts/error_layout.json.erb (Duration: 2.5ms | Allocations: 192)
2023-04-26T12:38:52 [I|app|401f63bd] Completed 500 Internal Server Error in 26ms (Views: 4.6ms | ActiveRecord: 1.8ms | Allocations: 2104)
2023-04-26T12:38:55 [I|app|8fbfb2fa] Started GET "/node/puppettest.mydomain.ltd?format=yml" for 172.30.0.25 at 2023-04-26 12:38:55 +0200
2023-04-26T12:38:55 [I|app|8fbfb2fa] Processing by HostsController#externalNodes as YML
2023-04-26T12:38:55 [I|app|8fbfb2fa]   Parameters: {"name"=>"puppettest.mydomain.ltd"}
2023-04-26T12:38:55 [I|app|8fbfb2fa]   Rendered text template (Duration: 0.1ms | Allocations: 1)
2023-04-26T12:38:55 [I|app|8fbfb2fa] Completed 200 OK in 55ms (Views: 1.0ms | ActiveRecord: 15.4ms | Allocations: 4547)
2023-04-26T12:38:55 [I|app|087e378d] Started POST "/api/config_reports" for 172.30.0.25 at 2023-04-26 12:38:55 +0200
2023-04-26T12:38:55 [I|app|087e378d] Processing by Api::V2::ConfigReportsController#create as JSON
2023-04-26T12:38:55 [I|app|087e378d]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2023-04-26T12:38:55 [I|app|087e378d] Scanning report with: Foreman::PuppetReportScanner
2023-04-26T12:38:55 [I|app|087e378d] Imported report for puppettest.mydomain.ltd in 123.1 ms, status refreshed in 30.6 ms
2023-04-26T12:38:55 [I|app|087e378d]   Rendered api/v2/config_reports/create.json.rabl (Duration: 6.6ms | Allocations: 812)
2023-04-26T12:38:55 [I|app|087e378d] Completed 201 Created in 197ms (Views: 10.9ms | ActiveRecord: 74.5ms | Allocations: 17227)
10

One other idea that comes to my mind:
Puppet 7 changed the default behavior of how facts (and maybe also node information) is stored on disk: Puppet release notes and following.
This has been fixed by Support JSON fact cache · theforeman/puppet-puppetserver_foreman@0fae2ed · GitHub in April 2021, but if this is an install without foreman-installer, you might have copied an old file. Or, if this was an upgrade, you might have missed updating the enc script.

The very noticable “incompatible marshal file format” error hints at some problems with your installation (that I don’t understand), but this is likely unrelated to this problem.

11

Ok. But this installation is installed with the foreman-installer only. And before update to 3.6.1 it was working.
No any other ideas?

12

I have taken another look at the logs you provided and I have found the inital 404 on the first puppetrun. I have probably overlooked it the first time I scrolled through your logs. Do you have “Create new host when facts are uploaded” enabled in Administer → Settings → Facts? I think this is enabled by default, so I assume you have this set to true.
What I assume is happening:

  • Puppet queries the ENC and fails since Foreman returns a 404 error (host not found)
  • The Puppetrun failes, but facts are still uploaded to Foreman, which causes a new host to be created (you can even see that in the Foreman log, guess I really did not look past that big stacktrace…)
  • Consecutive Puppetruns find the newly created host instead of the “real” one, which is why it works on the second and later runs, but with very bare ENC content

Can you check if you can find a second copy of your host of you set both organization and location to “Any”?
If my assumptions are correct, the real problem we need to solve is why your Foreman does not find the host on the original query. You should be able to reproduce the situation of the original Puppetrun if you delete the second, barebone copy, if it exists.

13

Hi,

the setting Facts is true. I saw, the hosts was not mapped to the default location and organisation, but settings are correct. Thus should be one issue. the new is that I can find a .yaml File under /opt/puppetlabs/server/data/puppetserver/yaml/foreman/puppettest.mydomain.ltd.yaml but not the second file like puppettest.mydomain.ltd-push-facts.yaml

The log output from puppet server:

2023-05-02T09:03:15.026+02:00 INFO  [qtp901154776-61740] [puppetserver] Puppet 'replace_facts' command for puppettest.mydomain.ltd submitted to PuppetDB with UUID bd374425-cea4-4457-9663-b45da4331cdb
2023-05-02T09:03:15.717+02:00 INFO  [qtp901154776-61740] [puppetserver] Puppet Compiled catalog for puppettest.mydomain.ltd in environment production in 0.09 seconds
2023-05-02T09:03:15.717+02:00 INFO  [qtp901154776-61740] [puppetserver] Puppet Caching catalog for puppettest.mydomain.ltd
2023-05-02T09:03:15.755+02:00 INFO  [qtp901154776-61740] [puppetserver] Puppet 'replace_catalog' command for puppettest.mydomain.ltd submitted to PuppetDB with UUID 1fe9fcaf-bd39-40cd-98c9-8acb98fcefd8
2023-05-02T09:03:16.635+02:00 INFO  [qtp901154776-61700] [puppetserver] Puppet 'store_report' command for puppettest.mydomain.ltd submitted to PuppetDB with UUID 92ecf569-50be-4faa-a204-3fadbdb7b02d

Foreman:

2023-05-02T09:03:15 [I|app|2656ba50] Started GET "/node/puppettest.mydomain.ltd?format=yml" for 172.30.0.25 at 2023-05-02 09:03:15 +0200
2023-05-02T09:03:15 [I|app|2656ba50] Processing by HostsController#externalNodes as YML
2023-05-02T09:03:15 [I|app|2656ba50]   Parameters: {"name"=>"puppettest.mydomain.ltd"}
2023-05-02T09:03:15 [I|app|2656ba50]   Rendered text template (Duration: 0.1ms | Allocations: 1)
2023-05-02T09:03:15 [I|app|2656ba50] Completed 200 OK in 136ms (Views: 1.8ms | ActiveRecord: 31.0ms | Allocations: 11231)
2023-05-02T09:03:16 [I|app|127595ba] Started POST "/api/config_reports" for 172.30.0.25 at 2023-05-02 09:03:16 +0200
2023-05-02T09:03:16 [I|app|127595ba] Processing by Api::V2::ConfigReportsController#create as JSON
2023-05-02T09:03:16 [I|app|127595ba]   Parameters: {"config_report"=>"[FILTERED]", "apiv"=>"v2"}
2023-05-02T09:03:16 [I|app|127595ba] Scanning report with: Foreman::PuppetReportScanner
2023-05-02T09:03:16 [I|app|127595ba] Imported report for puppettest.mydomain.ltd in 148.5 ms, status refreshed in 64.7 ms
2023-05-02T09:03:16 [I|app|127595ba]   Rendered api/v2/config_reports/create.json.rabl (Duration: 11.8ms | Allocations: 812)
2023-05-02T09:03:16 [I|app|127595ba] Completed 201 Created in 291ms (Views: 20.6ms | ActiveRecord: 91.3ms | Allocations: 18493)

node.rb:

---
parameters:
  foreman_config_groups: []
  puppetmaster: puppet.****
  puppet_ca: puppet.****
  foreman_env: production
  foreman_hostname: puppettest.mydomain.ltd
  foreman_fqdn: puppettest.mydomain.ltd
  root_pw: 
  foreman_subnets: []
  foreman_interfaces:
  - ip: 
    ip6: ''
    mac: 
    name: puppettest.mydomain.ltd
    attrs: {}
    virtual: false
    link: true
    identifier: ''
    managed: true
    primary: true
    provision: true
    subnet: 
    subnet6: 
    tag: 
    attached_to: 
    type: Interface
  location: Default Location
  location_title: Default Location
  organization: Default Organization
  organization_title: Default Organization
  owner_name: *******
  owner_email: ****@****
  ssh_authorized_keys:
  - ssh-rsa ****
  foreman_users:
    ****:
      firstname: ****
      lastname: ****
      mail: ****@****
      description: ''
      fullname: *******
      name: ****
      ssh_authorized_keys:
      - type: ssh-rsa
        key: ****
        comment: ****
  host_packages: ''
  host_registration_insights: false
  host_registration_remote_execution: true
classes: {}
environment: production

02
021331×408 24.9 KB

03
031868×769 110 KB

14

no idea? nothing?

15

Just returned from vacation, so I could not reply earlier, but I am out of ideas.
To me, everything looks correct.

16

I also having this same issue after upgrading to 3.6.1. I noticed that OS facts and some custom facts were not updated at all. Also the same errors in the logs.

I also have another possible side affect (might be related?). I changed a host smart parameter value from ‘stable’ to ‘unstable’. The Puppet ENC in Foreman shows me unstable. But when running the agent and printing the smart class parameter, it shows me it has a stable value. Also the catalog file in /var/lib/puppet/client_data/catalog/i-libre.linqhost.net.json is showing me stable.

17

Update:
After restarting Puppetserver the error seems to be away. The other issue has nothing todo with it.

18

I am also having this issue. I have a foreman server (foreman.local) and a foreman smart proxy (foreman-proxy.local) with puppetserver running on it.

If I point the client to foreman.local I then the client pushes the facts with information. But if i point it to the proxy i get the facts empty. I could give more info if necessary

19

Hey @twilms. I found something that worked for me.
In the file /etc/puppetlabs/puppet/foreman.yaml try setting :fact_extension: json

I think puppet 7 uses json.
I was just reading the node.rb and noticed that all the facts are in /opt/puppetlabs/server/data/puppetserver/server_data/facts/ but if the fact_extension is set to yaml (the default) then it will try to look in /opt/puppetlabs/server/data/puppetserver/yaml/facts/ which is empty.

Hopefully that helps you