Added two new puppetmasters then facts and reports uploads fail with Error 500 randomly

We recently added two new puppetmasters then facts and reports uploads fail with Error 500 randomly.

2022-02-15T10:44:37.372Z WARN  [qtp345670706-37] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: During the fact upload the server responded with: 500 Internal Server Error. Error is ignored and the execution continues.
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error or
misconfiguration and was unable to complete
your request.</p>
<p>Please contact the server administrator at
 [no address given] to inform them of the time this error occurred,
 and the actions you performed just before this error.</p>
<p>More information about this error may be available
in the server error log.</p>

Looking for help how to better debug this to find the root cause. This is only happening to the two new puppetmasters built on two different networks(firewalls seem to be fine).We already have like 10 more on other regions working without any issues. The errors are random, sometimes puppet runs sometimes it takes very long and foreman comes back with that Error 500. I ran simple curl to the foreman URL from them and it responds fine every time. Again I’m looking for help on how to know more what’s causing foreman to take that long an error so randomly.


Expected outcome:
puppet run without errors

Foreman and Proxy versions:
Foreman and Proxy plugin versions:
Distribution and version:
Centos 7
Other relevant data:

I’ve solved this. Not related to foreman at all but a network MTU problem.