Cannot retrieve catalog. Error 400 on server. Failed when searching for node

Hi all,

I'm posting this here because I no longer know what else to check in order
to resolve my issue. I have a multi puppet master + foreman setup running
on four CentOS 6 hosts- three puppet masters and a foreman host. Foreman
acts as my ENC and CA. Of the 157 hosts I manage with this system, only
three are having issues when checking in with the master. All three display
the error below:

Error: Could not retrieve catalog from remote server: Error 400 on SERVER:
Failed when searching for node myhost.domain.foo.bar: Failed to
find myhost.domain.foo.bar via exec: Execution of
'/etc/puppet/node.rb myhost.domain.foo.bar' returned :
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

Running */etc/puppet/node.rb myhost.domain.foo.bar *from the master yields
the following result:

··· --- classes: bash: common_packages: cron: customfacts: fail2ban: ntp: puppet: resolvconf: ssh: sysctl: syslog_ng: yum_repos: zabbix: parameters: puppetmaster: puppet-L-1.domain.foo.bar hostgroup: *redacted* root_pw: foreman_env: production owner_name: Admin User owner_email: *redacted* foreman_subnets: [] foreman_interfaces: - mac: ip: type: Interface name: myhost.domain.foo.bar attrs: {} virtual: false link: true identifier: '' managed: true primary: true provision: true subnet: environment: production

I’ve tried removing the host from foreman completely. I’ve even removed and
reinstalled the puppet agent completely. Foreman’s production logs don’t
report anything ominous however, the puppet master’s apache logs do show
this:

172.18.224.21 - - [06/Jul/2016:14:06:55 +0200] “GET
/production/node/myhost.domain.foo.bar?transaction_uuid=7c046a4a-705a-4e64-905c-c10d2fe3cdb2&fail_on_404=true
HTTP/1.1” 200 9728 “-” "-"
172.18.224.21 - - [06/Jul/2016:14:06:58 +0200] “POST
/production/catalog/myhost.domain.foo.bar HTTP/1.1” 400 205 “-” "-"
172.18.224.21 - - [06/Jul/2016:14:07:59 +0200] “PUT
/production/report/myhost.domain.foo.bar HTTP/1.1” 200 17 “-” “-”

Has anyone else bumped into this issue?