I have a new node. When I try to execute Puppet, the Puppetserver returns an error. Puppet seems to work fine on other clients.
web3 # puppet agent --test
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Error 500 on SERVER: Server Error: Failed to find web3.example.org via exec: Execution of '/etc/puppetlabs/puppet/node.rb web3.example.org' returned 1:
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error: Failed when searching for node web3.example.org: Failed to find web3.example.org via exec: Execution of '/etc/puppetlabs/puppet/node.rb web3.example.org' returned 1:
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
On the server side, /var/log/puppetlabs/puppetserver/puppetserver.log returns a super long stacktrace. But the key part seems to be this:
2019-11-14 15:06:12,512 WARN [qtp2056185030-3186] [c.p.p.ShellUtils] Executed an external process which logged to STDERR: During fact upload occured an exception: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
/etc/puppetlabs/puppet/node.rb:414: warning: constant ::TimeoutError is deprecated
Serving cached ENC: Could not send facts to Foreman: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate in certificate chain)
Unable to read from Cache file: No such file or directory @ rb_sysopen - /opt/puppetlabs/server/data/puppetserver/yaml/foreman/web3.example.org.yaml
2019-11-14 15:06:12,515 ERROR [qtp2056185030-3186] [puppetserver] Puppet Server Error: Failed to find web3.example.org via exec: Execution of '/etc/puppetlabs/puppet/node.rb web3.example.org' returned 1:
file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/execution.rb:56:in `execute'
file:/opt/puppetlabs/server/apps/puppetserver/puppet-server-release.jar!/puppetserver-lib/puppet/server/execution.rb:14:in `initialize_execution_stub'
org/jruby/RubyProc.java:281:in `call'
...
...
...
And indeed, that directory is missing my server. In addition, I note that the timestamps on all of these files is quite old.
root@puppet:~# ls -lA /opt/puppetlabs/server/data/puppetserver/yaml/foreman/
total 120
-rw-r----- 1 puppet puppet 112 Mar 25 2019 puppet.example.org-push-facts.yaml
-rw-r----- 1 puppet puppet 1940 Mar 25 2019 puppet.example.org.yaml
-rw-r----- 1 puppet puppet 112 Mar 25 2019 webtest07.example.org-push-facts.yaml
-rw-r----- 1 puppet puppet 2850 Mar 25 2019 webtest07.example.org.yaml
-rw-r----- 1 puppet puppet 112 Mar 25 2019 webtest08.example.org-push-facts.yaml
-rw-r----- 1 puppet puppet 2850 Mar 25 2019 webtest08.example.org.yaml
-rw-r----- 1 puppet puppet 112 Mar 25 2019 webtest09.example.org-push-facts.yaml
-rw-r----- 1 puppet puppet 2850 Mar 25 2019 webtest09.example.org.yaml
-rw-r----- 1 puppet puppet 112 Mar 25 2019 webtest10.example.org-push-facts.yaml
-rw-r----- 1 puppet puppet 4001 Mar 25 2019 webtest10.example.org.yaml
-rw-r----- 1 puppet puppet 112 Mar 25 2019 webtest11.example.org-push-facts.yaml
-rw-r----- 1 puppet puppet 4001 Mar 25 2019 webtest11.example.org.yaml
-rw-r----- 1 puppet puppet 112 Mar 25 2019 webtest12.example.org-push-facts.yaml
-rw-r----- 1 puppet puppet 4001 Mar 25 2019 webtest12.example.org.yaml
-rw-r----- 1 puppet puppet 112 Mar 25 2019 webtest13.example.org-push-facts.yaml
-rw-r----- 1 puppet puppet 4477 Mar 25 2019 webtest13.example.org.yaml
-rw-r----- 1 puppet puppet 112 Mar 25 2019 webtest14.example.org-push-facts.yaml
-rw-r----- 1 puppet puppet 4477 Mar 25 2019 webtest14.example.org.yaml
root@puppet:~#
This isn’t happening on other nodes.
Expected outcome:
I expected the Puppet agent to communicate to the server and download its configuration.
The Foreman server doesn’t seem to be putting my server configuration into the same location as used by the other servers.
I am having this exact same issue. Existing nodes are working as expect.
I am adding a new node this morning and as you very nicely put above getting the same error:
Warning: Unable to fetch my node definition, but the agent run will continue:
My config is the same as yours above.
I am running a slightly old puppet server:
[root@puppet ~]# rpm -qa | grep puppetserver
puppetserver -5.3.3-1.el7.noarch
Also over the last 48hrs facts and reports have stopped reaching foreman server. I check the existing node agents, they are successfully running very 30 minutes still.
Have a look here, it’s not a fix but it does shed some light on the situation: Error 500 Unable to fetch my node definition
I did what this person did, copy an existing node’s .yaml and name it according to the new node you are trying to add. You can then do a successful ‘puppet agent -t’ from the new node.
Whether or not that gives the wrong facts to the new node or not, I am unsure. I tried creating a blank.yaml and that didn’t work.
I also tried copying the new.node.yaml from /opt/puppetlabs/server/data/puppetserver/yaml/facts to /opt/puppetlabs/server/data/puppetserver/yaml/foreman/ but for whatever reason that resulted in error, also.
it looks like you have some sort of certificate problem with your Puppet servers.
From the provided log entries, the foreman ENC script (/etc/puppetlabs/puppet/node.rb) exits with an SSL exception for self-signed certificates. Since that script also handles report and fact upload, that would fit the described problem. Existing nodes probably keep working because Puppet’s own cache for node definitions ( /opt/puppetlabs/server/data/puppetserver/yaml/foreman/existing.node.yaml) already has a cached file for those systems. It is likely that existing systems will stop working once that cache is expired.
You should both take a look at the node.rb config file (/etc/puppetlabs/puppet/foreman.yaml) and check if the SSL configuration there points to the correct certificates. In my case (we have Katello installed) the certificates are in the form of /etc/pki/katello/puppet/puppet_client*. Without Katello, the path should be different and (I believe) point somewhere to /etc/puppetlabs/puppet/ssl/.
An easy check if the SSL config in /etc/puppetlabs/puppet/foreman.yaml is messed up would be with openssl: openssl s_client -connect yourforeman.example.com:443 -CAfile /path/to/cafile/from/config.crt </dev/null
If that does not give you “Verify return code: 0 (ok)” at the end, your SSL config is definetly messed up.
You can look for the correct certificates yourself and set them up manually. Alternatively, foreman-installer should correct these settings if you rerun it (be aware that other manual changes to configs managed by the installer will also be overridden, run the installer with “-v --noop” first in case you are unsure if this might be a problem).
Addtional note: I just talked to a collegue who told me he had this happen on a RedHat Satellite 6.5 and hat to regenerate the Satellite’s certificates in ordner to get this resolved. So it would probably be a good idea to also check your Foreman’s certificates for corruption just in case.
Thanks let me look at how to regenerate my certificates. My Puppet Master + CA is on one host, which is running foreman-proxy. This is where I think I should try to regenerate certs.
Then I have foreman running on a dedicated host. I also have Puppet Master running on my foreman host, but I don’t use it.
It was related to SSL, I found some things in the logs.
I decided to revert both my puppet master and foreman hosts to a Snapshot and attempt the install of both again.
I suspect it is related to my install options.
What I am trying to achieve is an existing Puppet Master to remain CA and install foreman-proxy.
Foreman to install and not be CA and if not necessary - not even run a Puppet Master.
We discovered that the :ssl_ca: parameter in /etc/puppetlabs/puppet/foreman.yaml is responsible for the problem. We comment it out, and it works. Uncomment it, and it breaks:
Note that the ssl_ca can be set with the installer using --puppet-server-foreman-ssl-ca. Removing the option disables verification of the CA and setting this to the correct CA will increase security.
I had the same issue. This thread set me light for the solution.
My usecase I had 2 servers. 1 is the foreman master server which is the CA. Second server was a smart proxy with puppet master talking to foreman.
Fix: The fix was setting the same ssl_ca (/etc/puppetlabs/puppet/foreman.yaml) parameter on the second server. I ensured that this parameter is the same for both foreman master and the foreman-proxy. It fixed the above mentioned error.
