Hi all,
after upgrading foreman and smart proxies from 2.3.5 to 2.5.2 reports are not forwarded to foreman anymore.
puppet works fine between client and SP.
can someone give me a hint where I can start to search?
how are the reports are forwarded to foreman?
Checl all the configurations steps from our manual chapter 3.5.4 Puppet Reports, if all seems ok, let us know. It could be an SSL misconfiguration, network issue, bug. We’d need some log files to determine.
Hi,
thx for the replay…
I checked all the configs and hints from the manual, but it didn’t help me.
I went back to an older snapshot of the smart-proxy, before upgrade, and foreman also don’t get any reports. So for me it seems, that the problem is on foreman site. I can’t find any errors in logs, not on foreman and not on SP.
do you have any idea what to check?
we can exclude network issues, it’s for sure ok.
which logs could help you?
thank you!
Alright, so first start watching the Foreman logs by tail -f ~foreman/log/production.log. Also in another terminal, watch for changes in Smart Proxy logs, tail -f ~foreman-proxy/logs/proxy.log. Then in the third terminal, start the puppet agent somewhere. Let’s see if you see some request in Proxy logs. If not, something is wrong with the reporter or its configuration. If you do see something, post it here.
Hi,
on client:
$ puppet agent -t
Info: Using configured environment ‘production’
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for demunlw310300.xyzcorp.lan
Info: Applying configuration version ‘1628507787’
Notice: Applied catalog in 0.16 seconds
on proxy only old logs from startup:
$ tail -f ~foreman-proxy/logs/proxy.log
2021-08-09T06:30:13 [I] Logging file reopened via USR1 signal
2021-08-09T09:20:47 a3526036 [I] Started GET /version
2021-08-09T09:20:47 a3526036 [I] Finished GET /version with 200 (0.64 ms)
on foreman nothing related:
sorry, not really helpful 
On your puppetserver, run grep foreman /etc/puppetlabs/puppet/puppet.conf
do you see line reports = foreman?
Anything interesting in /var/log/puppetlabs/puppetserver/puppetserver.log?
Is everything set correctly in /etc/puppetlabs/puppet/foreman.yaml
Do you see /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb on the filesystem?
grep foreman /etc/puppetlabs/puppet/puppet.conf
reports = log, foreman
tail /var/log/puppetlabs/puppetserver/puppetserver.log
2021-08-09T13:38:39.545Z INFO [qtp955639373-42] [puppetserver] //dewansl1013.xyzcorp.lan/Puppet Applied catalog in 0.20 seconds
2021-08-09T13:39:34.732Z INFO [qtp955639373-40] [puppetserver] Puppet Compiled catalog for demunlw310180.xyzcorp.lan in environment production in 0.34 seconds
2021-08-09T13:39:35.303Z INFO [qtp955639373-40] [puppetserver] //demunlw310180.xyzcorp.lan/Puppet Applied catalog in 0.14 seconds
2021-08-09T13:42:04.033Z INFO [qtp955639373-44] [puppetserver] Puppet Compiled catalog for deephsl1000.xyzcorp.lan in environment production in 0.45 seconds
2021-08-09T13:42:05.004Z INFO [qtp955639373-44] [puppetserver] //deephsl1000.xyzcorp.lan/Puppet Applied catalog in 0.71 seconds
2021-08-09T13:51:23.322Z INFO [qtp955639373-45] [puppetserver] Puppet Compiled catalog for dewoblw310001.xyzcorp.lan in environment production in 0.36 seconds
2021-08-09T13:51:23.566Z INFO [qtp955639373-40] [puppetserver] //dewoblw310001.xyzcorp.lan/Puppet Applied catalog in 0.13 seconds
2021-08-09T13:57:26.538Z INFO [qtp955639373-39] [puppetserver] Puppet Compiled catalog for demunlw310321.xyzcorp.lan in environment production in 0.35 seconds
2021-08-09T13:57:26.723Z INFO [qtp955639373-45] [puppetserver] //demunlw310321.xyzcorp.lan/Puppet Applied catalog in 0.11 seconds
2021-08-09T13:58:00.426Z INFO [qtp955639373-46] [puppetserver] Puppet Compiled catalog for demzksl1037.xyzcorp.lan in environment production in 0.37 seconds
cat /etc/puppetlabs/puppet/foreman.yaml
:url: “https://atgrzsl8135.xyzcorp.lan”
:ssl_ca: “/etc/pki/katello/puppet/puppet_client_ca.crt”
:ssl_cert: “/etc/pki/katello/puppet/puppet_client.crt”
:ssl_key: “/etc/pki/katello/puppet/puppet_client.key”
:puppetdir: “/opt/puppetlabs/server/data/puppetserver”
:puppetuser: “puppet”
:facts: true
:fact_extension: “yaml”
:timeout: 60
:report_timeout: 60
:threads: null
ls -las /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb
8 -rw-r–r–. 1 root root 6806 Apr 26 08:31 /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb
Since you have also log report handler, you should see the report data in syslog, take a look if you see something there.
Since everything looks correctly configured, I’d check the certificates are correct. Since you don’t see anything in the proxy log, if puppet server actually runs the foreman report handler, it probably can’t connect for some reason. Try checking the certificates expirations by
openssl x509 -in /etc/pki/katello/puppet/puppet_client_ca.crt -dates
openssl x509 -in /etc/pki/katello/puppet/puppet_client.crt -dates
Otherwise, I’d start debugging /opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb. Given there’s no error from the reporter, there’s still a chance puppetserver does not run it at all. But I have no idea why. Did you also updated puppet when you upgraded your smart proxy?
Thank you so much for your help…
for me, it looks like the issue is on foreman site !?
in syslog, I can see only the own reports…
tail -f messages | grep puppet
Aug 10 11:17:55 dedcfsl8135 puppet-agent[1926]: Using configured environment ‘production’
Aug 10 11:17:55 dedcfsl8135 puppet-agent[1926]: Retrieving pluginfacts
Aug 10 11:17:55 dedcfsl8135 puppet-agent[1926]: Retrieving plugin
Aug 10 11:17:56 dedcfsl8135 puppet-agent[1926]: Retrieving locales
Aug 10 11:17:56 dedcfsl8135 puppet-agent[1926]: Loading facts
Aug 10 11:17:59 dedcfsl8135 puppet-agent[1926]: Caching catalog for dedcfsl8135.xyzcorp.lan
Aug 10 11:17:59 dedcfsl8135 puppet-agent[1926]: Applying configuration version ‘1628587078’
Aug 10 11:17:59 dedcfsl8135 puppet-agent[1926]: Applied catalog in 0.29 seconds
certificates looking good:
openssl x509 -in /etc/pki/katello/puppet/puppet_client_ca.crt -dates
notBefore=Apr 15 11:21:50 2015 GMT
notAfter=Apr 15 11:31:48 2055 GMT
-----BEGIN CERTIFICATE-----
…
openssl x509 -in /etc/pki/katello/puppet/puppet_client.crt -dates
notBefore=Apr 19 08:27:48 2021 GMT
notAfter=Jan 18 08:27:50 2038 GMT
-----BEGIN CERTIFICATE-----
…
on foreman:
rpm -qa | grep puppet
atgrzsl8135.xyzcorp.lan-puppet-client-1.0-1.noarch
puppet-agent-6.24.0-1.el7.x86_64
puppetserver-6.16.1-1.el7.noarch
puppet-agent-oauth-0.5.5-1.el7.noarch
puppet6-release-6.0.0-14.el7.noarch
on SP:
rpm -qa | grep puppet
puppetserver-6.16.1-1.el7.noarch
puppet-agent-oauth-0.5.5-1.el7.noarch
puppet-agent-6.24.0-1.el7.x86_64
puppet6-release-6.0.0-14.el7.noarch
dedcfsl8135.xyzcorp.lan-puppet-client-1.0-1.noarch
I found following in foreman GUI → Smart Proxies → Puppet CA:
Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA certificates ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy https://dedcfsl8135.xyzcorp.lan:9090/puppet/ca)
Failure: ERF50-5345 [Foreman::WrappedException]: Unable to connect ([ProxyAPI::ProxyException]: ERF12-5356 [ProxyAPI::ProxyException]: Unable to get PuppetCA certificates ([RestClient::NotAcceptable]: 406 Not Acceptable) for proxy https://dedcfsl8135.xyzcorp.lan:9090/puppet/ca)
Then it feels more like the SSL certificates misconfiguration, try looking at Foreman-Proxy with Puppet-CA: Unable to get PuppetCA certificates: 406 Not Acceptable for proxy
I wouldn’t be surprised if the certs on Puppet server or Smart Proxy side are not trusted on the Foreman side. That would explain also the reason why reports are not delivered. Did you by any chance tried to modify Foreman’s SSL certificates?
at least it wasn’t my intention… can this happened by the update?
what can I try now?
create new certificates?
Update shouldn’t touch this at all. I’d start by trying to curl the smart proxy API endpoint using the certs you find under the settings:
- SSL certificate
- SSL CA file
- SSL private key
You need to figure out whether that works. You can check if the issuer of the SSL certificate is the same CA that Smart Proxy uses.
I hope I don’t mislead you with this hint, but it really feels as some SSL misconfiugration. Perhaps others have more ideas.