Updating puppet from 3.8.7 to 4.8.1 with Foreman 1.12.4 on EL6.8

I'm doing my best to follow all the instructions on the puppet and foreman
websites and wikis. I believe I have most of it working - the webUI is
good, the puppetserver is talking to clients. The main issue is that the
clients get:

[root@lnx246 ~]# puppet agent -t
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: {"message":"Server Error: Failed to find lnx246 via exec: Execution of '/etc/puppetlabs/puppet/node.rb lnx246' returned 1: ","issue_kind":"RUNTIME_ERROR"}
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: {"message":"Server Error: Failed when searching for node lnx246: Failed to find lnx246.classe.cornell.edu via exec: Execution of '/etc/puppetlabs/puppet/node.rb lnx246' returned 1: ","issue_kind":"RUNTIME_ERROR"}
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Running node.rb on the CLI gives:
sudo -u puppet /etc/puppetlabs/puppet/node.rb puppet.node.fqdn
Permission denied - /etc/puppetlabs/puppet/yaml

There is no file called yaml, nor am I at all certain why node.rb would be
looking for that file. I feel like there's a small piece of configuration
that is needed to get it all working right. I did set in the foreman proxy
settings the current version of puppet, 4.8.1 replacing 3.8.7 so foreman
should know it's a newer version… The clients are still 3.8.7, but I put
the legacy support flags in auth.conf as suggested. However, given that the
agents can get far enough to request an ENC lookup, I don't think it's a
failure at the level of v3 agent -> v4 server and I'm pretty sure that also
means the server can find the ssl certs. So I think this is related to the
ENC config for foreman's node.rb . . .

Help?

I fixed this by setting the correct puppet cache directory for foreman. Now
I'm on to another issue, where I cannot import puppet classes because of:

ERF12-4115 406 Not Acceptable … We've tried changing the auth.conf rules
to no effect - that shows me a wiki page that has me check
/var/log/foreman-proxy/proxy.log
which shows:
ERROR – : Failed to show puppet classes: 403 {"message":"Not Authorized:
Forbidden request: /puppet/v3/resource_type/*
[search]","issue_kind":"RUNTIME_ERROR"}

The fix for this ended up being disabling the legacy auth.conf and just
using the new format with the suggested foreman settings from their guide.
V3 agents check in successfully.

Final take away - I think that anyone else trying this after us should
mostly follow the foreman wiki and ignore a lot of the server / master
stuff from puppet directly. Of course you do need to refer to puppet
guidelines for fixing the manifests. Future parser did not find a lot of
issues for us so we ended up fixing them after the upgrade - carefully
working through the puppet guidelines and not assuming the agents will
error would also help.

After Updating puppet server from 3.8.7 to 4.10 on Foreman 1.15.1.
I am getting the same error messages as you did on client machines, 'Error:
Could not retrieve catalog from remote server: Error 500 on SERVER…'

I
followed Upgrading from Puppet 3 to 4 - Foreman

My puppet.conf has vardir = /opt/puppetlabs/puppet/cache

And I copied /var/lib/puppet/foreman_cache_data to
/opt/puppetlabs/puppet/cache/foreman_cache_data

You said, 'I fixed this by setting the correct puppet cache directory for
foreman'. Can you please let me know what exactly you did?

Thanks