Cannot retrieve catalog. Error 400 on server. Failed when searching for node

Hi all,

I'm posting this here because I no longer know what else to check in order
to resolve my issue. I have a multi puppet master + foreman setup running
on four CentOS 6 hosts- three puppet masters and a foreman host. Foreman
acts as my ENC and CA. Of the 157 hosts I manage with this system, only
three are having issues when checking in with the master. All three display
the error below:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
Failed when searching for node Failed to
find via exec: Execution of
'/etc/puppet/node.rb' returned :
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Running */etc/puppet/node.rb *from the master yields
the following result:

··· --- classes: bash: common_packages: cron: customfacts: fail2ban: ntp: puppet: resolvconf: ssh: sysctl: syslog_ng: yum_repos: zabbix: parameters: puppetmaster: hostgroup: *redacted* root_pw: foreman_env: production owner_name: Admin User owner_email: *redacted* foreman_subnets: [] foreman_interfaces: - mac: ip: type: Interface name: attrs: {} virtual: false link: true identifier: '' managed: true primary: true provision: true subnet: environment: production

I’ve tried removing the host from foreman completely. I’ve even removed and
reinstalled the puppet agent completely. Foreman’s production logs don’t
report anything ominous however, the puppet master’s apache logs do show
this: - - [06/Jul/2016:14:06:55 +0200] “GET
HTTP/1.1” 200 9728 “-” "-" - - [06/Jul/2016:14:06:58 +0200] “POST
/production/catalog/ HTTP/1.1” 400 205 “-” "-" - - [06/Jul/2016:14:07:59 +0200] “PUT
/production/report/ HTTP/1.1” 200 17 “-” “-”

Has anyone else bumped into this issue?