No host puppet reports since Foreman/Katello upgrade to 2.5/4.1

Problem: I upgraded Foreman/Katello last week from 2.4/4.0 to 2.5/4.1.
I just noticed today that since the upgrade, there are zero puppet reports.
I run ‘puppet agent -t’ on a few servers, and it is running as it should, receiving manifests and applying changes. But the last reports I have are from 7 days ago.
Ansible roles ARE providing reports as expected. It seems to only be puppet that isn’t providing reports.

I checked puppetserver.conf and puppetserver.conf.rpmsave, and there appear to be a couple changes, but seemingly unrelated to anything for reports.

This is not a new server…it’s been running for 4 years

Expected outcome: Reports for each host every 30 minutes

Foreman and Proxy versions:
Foreman 2.5.3
Katello 4.1.3
puppetserver-5.3.16-1.el7.noarch

Distribution and version:
CentOS Linux release 7.9.2009

Other relevant data: From production.log during ‘puppet agent -t’ run:

2021-09-22T15:35:19 [I|app|d4bb8a13] Started GET "/node/myhost.example.com?format=yml" for 10.1.2.3 at 2021-09-22 15:35:19 +0000
2021-09-22T15:35:19 [I|app|d4bb8a13] Processing by HostsController#externalNodes as YML
2021-09-22T15:35:19 [I|app|d4bb8a13]   Parameters: {"name"=>"myhost.example.com"}
2021-09-22T15:35:20 [I|app|d4bb8a13]   Rendering text template
2021-09-22T15:35:20 [I|app|d4bb8a13]   Rendered text template (Duration: 0.0ms | Allocations: 1)
2021-09-22T15:35:20 [I|app|d4bb8a13] Completed 200 OK in 371ms (Views: 0.7ms | ActiveRecord: 53.8ms | Allocations: 133108)
2021-09-22T15:35:22 [I|app|181bdfe9] Started POST "/api/hosts/facts" for 10.1.2.3 at 2021-09-22 15:35:22 +0000
2021-09-22T15:35:22 [I|app|181bdfe9] Processing by Api::V2::HostsController#facts as JSON
2021-09-22T15:35:22 [I|app|181bdfe9]   Parameters: {"facts"=>"[FILTERED]", "name"=>"myhost.example.com", "certname"=>"myhost.example.com", "apiv"=>"v2", "host"=>{"certname"=>"myhost.example.com", "name"=>"myhost.example.com"}}
2021-09-22T15:35:23 [I|app|181bdfe9] Import facts for 'myhost.example.com' completed. Added: 0, Updated: 15, Deleted 0 facts
2021-09-22T15:35:23 [W|app|181bdfe9] Setting puppetrun has no definition, please define it before using
2021-09-22T15:35:23 [W|app|181bdfe9] Setting puppetrun has no definition, please define it before using
2021-09-22T15:35:23 [I|app|181bdfe9] Completed 201 Created in 855ms (Views: 4.3ms | ActiveRecord: 211.8ms | Allocations: 213506)
2021-09-22T15:35:23 [I|app|444b5b46] Started GET "/node/myhost.example.com?format=yml" for 10.1.2.3 at 2021-09-22 15:35:23 +0000
2021-09-22T15:35:23 [I|app|444b5b46] Processing by HostsController#externalNodes as YML
2021-09-22T15:35:23 [I|app|444b5b46]   Parameters: {"name"=>"myhost.example.com"}
2021-09-22T15:35:23 [I|app|444b5b46]   Rendering text template
2021-09-22T15:35:23 [I|app|444b5b46]   Rendered text template (Duration: 0.0ms | Allocations: 1)
2021-09-22T15:35:23 [I|app|444b5b46] Completed 200 OK in 367ms (Views: 0.9ms | ActiveRecord: 56.6ms | Allocations: 133303)
2021-09-22T15:35:25 [I|app|f1c6c2ff] Started POST "/api/hosts/facts" for 10.1.2.3 at 2021-09-22 15:35:25 +0000
2021-09-22T15:35:25 [I|app|f1c6c2ff] Processing by Api::V2::HostsController#facts as JSON

@mshade ,

Did you find a solution for this?

Im having the same problem after first upgrading Foreman/Katello from 2.4.x/4.0.x to 2.5.x/4.1.x followed by an upgrade from 2.5.x/4.1.x to 3.0.x/4.2.x.

‘puppet agent -t’ runs here also as it should be.

Even worse, after these 2 upgrades, all hosts have “Out of sync” for “Configuration”.

Support from anyone in the Foreman/Katello team is more than welcome.

Thanks in advance