Rebooted foreman server and Puppet Reporting stopped working

Problem: Reporting and Facts stopped updating on foreman.

Expected outcome: Continued reports and facts every 30 minutes, this had been working for approximately 4-5 days with out problem.

Foreman and Proxy versions: 1.23.0

Foreman and Proxy plugin versions: 1.23.0

Distribution and version: Centos 7.4

Other relevant data:
I am not really sure what would help? Which logs could I share here?

Just to give a bit more background info here, Foreman is on one host and my Puppet Master and CA are another running foreman-proxy. For almost 5 days the reporting and facts have been sent and updating in Foreman successfully. I wanted to add more RAM to the foreman host so I shut it down, the first time I can remember over the last 4-5 days.

Once It came backup I noticed the reports were getting older than 1hour in Foreman. After checking puppet runs in the other hosts they were still completing fine - just the reports and facts not making it to foreman.

Thanks for any help, this was all working really good - I was about to put it into production.

Maybe it’s a stupid question:
Have you setup a recurring logic and is it working?
you can check it on Monitor -> Recurring Logics

Hi,

I would start by checking /var/log/foreman-proxy/proxy.log on your Puppet Proxy server and check there if you see any issues connecting to your main Foreman. Maybe also enable debugging on the Proxy if nothing interesting is logged.
Maybe you also want to check your /var/log/foreman/production.log on the Foreman server. The API requests for fact and report uploads should show up in there with the corresponding API endpoints.

Thanks for the tips.

This is the end of the production.log from Foreman Server.
https://pastebin.com/LLsERvJj

It doesn’t look like it is receiving the reports and facts from my Puppet Server. If I go to the start of this log you can see it.

For example like this:

“host”=>{“certname”=>“bld-a-0003-la.domain.com”, “name”=>“bld-a-0003-la.domain.com”}}
2019-11-13T03:52:18 [I|app|2f39d4d6] Import facts for ‘bld-a-0003-la.domain.com’ completed. Added: 0, Updated: 10, Deleted 0 facts
2019-11-13T03:52:18 [I|app|2f39d4d6] Completed 201 Created in 329ms (Views: 1.3ms | ActiveRecord: 76.8ms)
2019-11-13T03:52:18 [I|app|af55e981] Started GET “/node/bld-a-0003-la.domain.com?format=yml” for 10.64.80.226 at 2019-11-13 03:52:18 +0800
2019-11-13T03:52:18 [I|app|af55e981] Processing by HostsController#externalNodes as YML
2019-11-13T03:52:18 [I|app|af55e981] Parameters: {“name”=>“bld-a-0003-la.domain.com”}
2019-11-13T03:52:18 [I|app|af55e981] Rendering text template
2019-11-13T03:52:18 [I|app|af55e981] Rendered text template (0.0ms)
2019-11-13T03:52:18 [I|app|af55e981] Completed 200 OK in 89ms (Views: 0.4ms | ActiveRecord: 8.6ms)
2019-11-13T03:52:19 [I|app|2854cea3] Started POST “/api/config_reports” for 10.64.80.226 at 2019-11-13 03:52:19 +0800
2019-11-13T03:52:19 [I|app|2854cea3] Processing by Api::V2::ConfigReportsController#create as JSON
2019-11-13T03:52:19 [I|app|2854cea3] Parameters: {“config_report”=>"[FILTERED]", “apiv”=>“v2”}
2019-11-13T03:52:19 [I|app|2854cea3] Scanning report with: Foreman::PuppetReportScanner
2019-11-13T03:52:19 [I|app|2854cea3] Imported report for bld-a-0003-la.domain.com in 19.8 ms, status refreshed in 8.6 ms
2019-11-13T03:52:19 [I|app|2854cea3] Rendering api/v2/config_reports/create.json.rabl
2019-11-13T03:52:19 [I|app|2854cea3] Rendered api/v2/config_reports/create.json.rabl (5.1ms)
2019-11-13T03:52:19 [I|app|2854cea3] Completed 201 Created in 47ms (Views: 5.1ms | ActiveRecord: 15.3ms)
2019-11-13T03:53:05 [I|app|6ee4376f] Started GET “/node/bld-a-0016-la.domain.com?format=yml” for 10.64.80.226 at 2019-11-13 03:53:05 +0800
2019-11-13T03:53:05 [I|app|6ee4376f] Processing by HostsController#externalNodes as YML
2019-11-13T03:53:05 [I|app|6ee4376f] Parameters: {“name”=>“bld-a-0016-la.domain.com”}
2019-11-13T03:53:05 [I|app|6ee4376f] Rendering text template
2019-11-13T03:53:05 [I|app|6ee4376f] Rendered text template (0.0ms)
2019-11-13T03:53:05 [I|app|6ee4376f] Completed 200 OK in 97ms (Views: 0.4ms | ActiveRecord: 10.0ms)
2019-11-13T03:53:07 [I|app|e5f8222e] Started POST “/api/hosts/facts” for 10.64.80.226 at 2019-11-13 03:53:07 +0800
2019-11-13T03:53:07 [I|app|e5f8222e] Processing by Api::V2::HostsController#facts as JSON
2019-11-13T03:53:07 [I|app|e5f8222e] Parameters: {“facts”=>"[FILTERED]", “name”=>“bld-a-0016-la.domain.com”, “certname”=>“bld-a-0016-la.domain.com”, “apiv”=>“v2”,

This is the fail end of the foreman-proxy proxy.log from the Puppet Server. This looks ok to me, nothing obvious. https://pastebin.com/QsikFQz1

How do I turn debugging on?

I dont have that option under Monitor?

The proxy.log looks fine to me, too. You can turn debugging on via /etc/foreman-proxy/settings.yml, there is an option log_level that you can adjust and then restart foreman-proxy.
From looking at the production.log, it does not even look like there are any puppet runs taking place. Maybe try “tail -f production.log” and start a puppet run on one of your systems manually just to be sure that one actually happened.
Also if interest could be the content of /var/log/puppetlabs/puppetserver/puppetserver.log. Maybe there are any errors to be observed there.

It appears I have fixed it this morning, let me confirm what it was and report back…

Could you please share how you fixed this issue? I have same situation here. Thanks

@edw1111 do you use lets encrypt ?