Just to give a bit more background info here, Foreman is on one host and my Puppet Master and CA are another running foreman-proxy. For almost 5 days the reporting and facts have been sent and updating in Foreman successfully. I wanted to add more RAM to the foreman host so I shut it down, the first time I can remember over the last 4-5 days.
Once It came backup I noticed the reports were getting older than 1hour in Foreman. After checking puppet runs in the other hosts they were still completing fine - just the reports and facts not making it to foreman.
Thanks for any help, this was all working really good - I was about to put it into production.
I would start by checking /var/log/foreman-proxy/proxy.log on your Puppet Proxy server and check there if you see any issues connecting to your main Foreman. Maybe also enable debugging on the Proxy if nothing interesting is logged.
Maybe you also want to check your /var/log/foreman/production.log on the Foreman server. The API requests for fact and report uploads should show up in there with the corresponding API endpoints.
It doesn’t look like it is receiving the reports and facts from my Puppet Server. If I go to the start of this log you can see it.
For example like this:
“host”=>{“certname”=>“bld-a-0003-la.domain.com”, “name”=>“bld-a-0003-la.domain.com”}}
2019-11-13T03:52:18 [I|app|2f39d4d6] Import facts for ‘bld-a-0003-la.domain.com’ completed. Added: 0, Updated: 10, Deleted 0 facts
2019-11-13T03:52:18 [I|app|2f39d4d6] Completed 201 Created in 329ms (Views: 1.3ms | ActiveRecord: 76.8ms)
2019-11-13T03:52:18 [I|app|af55e981] Started GET “/node/bld-a-0003-la.domain.com?format=yml” for 10.64.80.226 at 2019-11-13 03:52:18 +0800
2019-11-13T03:52:18 [I|app|af55e981] Processing by HostsController#externalNodes as YML
2019-11-13T03:52:18 [I|app|af55e981] Parameters: {“name”=>“bld-a-0003-la.domain.com”}
2019-11-13T03:52:18 [I|app|af55e981] Rendering text template
2019-11-13T03:52:18 [I|app|af55e981] Rendered text template (0.0ms)
2019-11-13T03:52:18 [I|app|af55e981] Completed 200 OK in 89ms (Views: 0.4ms | ActiveRecord: 8.6ms)
2019-11-13T03:52:19 [I|app|2854cea3] Started POST “/api/config_reports” for 10.64.80.226 at 2019-11-13 03:52:19 +0800
2019-11-13T03:52:19 [I|app|2854cea3] Processing by Api::V2::ConfigReportsController#create as JSON
2019-11-13T03:52:19 [I|app|2854cea3] Parameters: {“config_report”=>"[FILTERED]", “apiv”=>“v2”}
2019-11-13T03:52:19 [I|app|2854cea3] Scanning report with: Foreman::PuppetReportScanner
2019-11-13T03:52:19 [I|app|2854cea3] Imported report for bld-a-0003-la.domain.com in 19.8 ms, status refreshed in 8.6 ms
2019-11-13T03:52:19 [I|app|2854cea3] Rendering api/v2/config_reports/create.json.rabl
2019-11-13T03:52:19 [I|app|2854cea3] Rendered api/v2/config_reports/create.json.rabl (5.1ms)
2019-11-13T03:52:19 [I|app|2854cea3] Completed 201 Created in 47ms (Views: 5.1ms | ActiveRecord: 15.3ms)
2019-11-13T03:53:05 [I|app|6ee4376f] Started GET “/node/bld-a-0016-la.domain.com?format=yml” for 10.64.80.226 at 2019-11-13 03:53:05 +0800
2019-11-13T03:53:05 [I|app|6ee4376f] Processing by HostsController#externalNodes as YML
2019-11-13T03:53:05 [I|app|6ee4376f] Parameters: {“name”=>“bld-a-0016-la.domain.com”}
2019-11-13T03:53:05 [I|app|6ee4376f] Rendering text template
2019-11-13T03:53:05 [I|app|6ee4376f] Rendered text template (0.0ms)
2019-11-13T03:53:05 [I|app|6ee4376f] Completed 200 OK in 97ms (Views: 0.4ms | ActiveRecord: 10.0ms)
2019-11-13T03:53:07 [I|app|e5f8222e] Started POST “/api/hosts/facts” for 10.64.80.226 at 2019-11-13 03:53:07 +0800
2019-11-13T03:53:07 [I|app|e5f8222e] Processing by Api::V2::HostsController#facts as JSON
2019-11-13T03:53:07 [I|app|e5f8222e] Parameters: {“facts”=>"[FILTERED]", “name”=>“bld-a-0016-la.domain.com”, “certname”=>“bld-a-0016-la.domain.com”, “apiv”=>“v2”,
The proxy.log looks fine to me, too. You can turn debugging on via /etc/foreman-proxy/settings.yml, there is an option log_level that you can adjust and then restart foreman-proxy.
From looking at the production.log, it does not even look like there are any puppet runs taking place. Maybe try “tail -f production.log” and start a puppet run on one of your systems manually just to be sure that one actually happened.
Also if interest could be the content of /var/log/puppetlabs/puppetserver/puppetserver.log. Maybe there are any errors to be observed there.