Out of sync for all hosts at once

Neeloj · October 7, 2021, 6:39am

Hi all,

All hosts have Out of sync at once since 2 days ago!

Expected outcome:

Foreman and Proxy versions:
3.0.1
Distribution and version:
Ubuntu 20.04

What can I do here or check ?

Thanks in advance

Neeloj

lzap · October 7, 2021, 8:35am

Contact your Foreman administrator!

The internet.

https://docs.theforeman.org/nightly/Administering_Red_Hat_Satellite/index-foreman-el.html#chap-Administering-Using_the_Red_Hat_Satellite_Content_Dashboard

https://docs.theforeman.org/nightly/Managing_Hosts/index-foreman-el.html#puppet_guide_setting_out_of_sync_time_for_puppet_managed_hosts_managing-hosts

Neeloj · October 7, 2021, 8:39am

thanks for your answer @lzap
I think below topic make the issue:

and when I type:
# hammer host list --search ‘last_report < “35 minutes ago” and status.enabled = true’
SSL certificate verification failed
Make sure you configured the correct URL and have the server’s CA certificate installed on your system.

The following configuration option were used for the SSL connection:
ssl_ca_file = /etc/letsencrypt/live/foreman.local/chain.pem

Make sure the location contains an unexpired and valid CA certificate for https://foreman.local.

lzap · October 7, 2021, 8:40am

What you describe is a different problem, create one thread per problem.

The out of sync can be fixed either by letting your hosts to report, or configuring the out of sync interval in settings, see our docs.

Neeloj · October 7, 2021, 8:42am

they are related to eachother, when I disable the SSL the reports works again!

I check the docs and the old topics here too which habe out of sync but it doesnt help!
Administer → settings → General & Puppet Tabs I change the interval for both it doesnt help!

lzap · October 7, 2021, 8:52am

You asked for out of sync, I suggest you to create a new topic describing exactly what you did (disable SSL what’s that?)

Neeloj · October 7, 2021, 8:59am

Thats right because I get :
grafik

And for me when I activate Letsencrypt Reports doent work!

anyway thanks for your answer.

ekohl · October 7, 2021, 10:47am

I would check /var/log/puppetlabs/puppetserver and what’s in the logs there
I ran into:

github.com/jruby/jruby

Jruby does not support the "old" LetsEncrypt chain after DST Root CA X3 has expired

opened 09:46AM - 01 Oct 21 UTC

closed 05:31PM - 27 Oct 21 UTC

laugmanuel

**Environment Information** Provide at least: * JRuby versions: * `jruby 9.2.17.0 (2.5.8) 2021-03-29 84d363da97 OpenJDK 64-Bit Server VM 25.302-b08 on 1.8.0_302-b08 +jit [linux-x86_64]` * `jruby 9.2.19.0 (2.5.8) 2021-06-15 55810c552b OpenJDK 64-Bit Server VM 25.302-b08 on 1.8.0_302-b08 +jit [linux-x86_64]` * `jruby 9.3.0.0 (2.6.8) 2021-09-17 85c20e780f OpenJDK 64-Bit Server VM 25.302-b08 on 1.8.0_302-b08 +jit [linux-x86_64]` * Operating system and platform: * OS: RedHat Enterprise Linux 7.9 and 8.4 * uname -a: `Linux <host> 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Tue Sep 7 07:07:31 EDT 2021 x86_64 x86_64 x86_64 GNU/Linux` Other relevant info you may wish to add: * Installed or activated gems: Only the `requests` gem was installed to verify the problem * Application/framework version (e.g. Rails, Sinatra): none * Environment variables: no specific Ruby env variables **Expected Behavior** Executing the following code should result in a successfull HTTP request: ```ruby require 'requests' require 'net/http' puts Net::HTTP.get_response(URI('https://helloworld.letsencrypt.org')) ``` **Actual Behavior** The code above results in: ```ruby OpenSSL::SSL::SSLError: certificate verify failed connect at /home/<user>/.rvm/rubies/jruby-9.3.0.0/lib/ruby/stdlib/net/http.rb:1018 do_start at /home/<user>/.rvm/rubies/jruby-9.3.0.0/lib/ruby/stdlib/net/http.rb:934 start at /home/<user>/.rvm/rubies/jruby-9.3.0.0/lib/ruby/stdlib/net/http.rb:923 start at /home/<user>/.rvm/rubies/jruby-9.3.0.0/lib/ruby/stdlib/net/http.rb:605 get_response at /home/<user>/.rvm/rubies/jruby-9.3.0.0/lib/ruby/stdlib/net/http.rb:481 <main> at test.rb:4 ```` If I run the smae code using native Ruby, the request works. **Additional information**: The used endpoint (`https://helloworld.letsencrypt.org`) presents the long chain (including `C=US, O=Let's Encrypt, CN=R3` and `C=US, O=Internet Security Research Group, CN=ISRG Root X1`). If the same code is ran against `https://valid-isrgrootx1.letsencrypt.org/`, where only `C=US, O=Let's Encrypt, CN=R3` is presented, the request works fine. So this confirms that: * Jruby does know about ISRG Root X1 and trusts it (otherwise the request against `https://valid-isrgrootx1.letsencrypt.org/` would fail) * Jruby does not trust the old chain because `DST Root CA X3` has expired This issue seems similar to https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

You can verify this by using openssl s_client -connect $HOSTNAME:$PORT. It should some something like:

Certificate chain
 0 s:/CN=$HOSTNAME
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1

If you see another part at the end of the chain that’s X3, you run into the problem. I tried certbot renew --force-renewal --preferred-chain "ISRG Root X1" but that didn’t work. For me the solution was to remove the last certificate from /etc/letsencrypt/live/$DOMAIN/chain.pem. I may run into a problem again when it’s renewed in a few months.

Neeloj · October 7, 2021, 11:23am

thansk for your answer @ekohl

You are right, I found this Error in the Log file:


2021-10-07T13:20:45.207+02:00 INFO  [qtp1660961896-34] [puppetserver] Puppet Computing checksum on string
2021-10-07T13:20:45.281+02:00 INFO  [qtp1660961896-2291] [puppetserver] Puppet Compiled catalog for svm-visusmain01.ad.kklbo.de in environment production in 0.12 seconds
2021-10-07T13:20:46.190+02:00 ERROR [qtp1660961896-34] [puppetserver] Puppet Report processor failed: Could not send report to Foreman at https://foreman.local/api/config_reports: certificate verify failed
["uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:1002:in `connect'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:924:in `do_start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/http.rb:913:in `start'", "uri:classloader:/META-INF/jruby.home/lib/ruby/stdlib/net/htt
p.rb:1465:in `request'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:69:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `blo
ck in processors'", "org/jruby/RubyArray.java:1809:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:30:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/pupp
et/indirector/report/processor.rb:14:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/indirection.rb:316:in `save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:199:in `do_save'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/
api/indirected_routes.rb:54:in `block in call'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/api/indirected_routes.rb:53:in `call'", "/opt/
puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:82:in `block in process'", "org/jruby/RubyArray.java:1809:in `each'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:81:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `proces
s'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/route.rb:88:in `process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:87:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:70:in `block in with_request_profiling'",
 "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler/around_profiler.rb:58:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/util/profiler.rb:51:in `profile'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:66:in `with_request_profiling'", "/opt/puppetlabs/
puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:86:in `block in process'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:93:in `respond_to_errors'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/network/http/handler.rb:85:in `process'", "uri:classloader:/puppetserver-li
b/puppet/server/master.rb:65:in `block in handleRequest'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/context.rb:62:in `override'", "/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet.rb:314:in `override'", "uri:classloader:/puppetserver-lib/puppet/server/master.rb:64:in `handleRequest'"]
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/reports/foreman.rb:75:in `process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:37:in `block in process'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:54:in `block in processors'
org/jruby/RubyArray.java:1809:in `each'
/opt/puppetlabs/puppet/lib/ruby/vendor_ruby/puppet/indirector/report/processor.rb:51:in `processors'

---
Certificate chain
 0 s:CN = foreman.local
   i:CN = Puppet CA: foreman.local
 1 s:CN = Puppet CA: foreman.local
   i:CN = Puppet CA: foreman.local

I dont have that at the end, but why I see the error!!!

Neeloj · October 7, 2021, 12:16pm

@ekohl You mean I should modify the chain.pem and remove last certificate right ?

ekohl · October 7, 2021, 12:21pm

Yes. However, it appears to be serving the Puppet CA certificate, not the Let’s Encrypt certificate. Was the configuration somehow reverted?

Neeloj · October 7, 2021, 12:25pm

Ok I have there two I’ll remove the last one.

No an and thats why Im confused why this happens!

As you can see above I dont have X3 or so!

I’ll test your suggestion and then Ill try with renew too! and give my feedback here

Neeloj · October 7, 2021, 12:27pm

This work for me too. But Ill try with renew too

damonmaria · October 11, 2021, 12:12am

I had the same problem posted here: Puppet reports suddenly stopped at same time for all hosts - #6 by damonmaria

In the end I added --preferred-chain="ISRG Root X1" to my letsencrypt certbot renew command.

Neeloj · October 13, 2021, 6:09am

thanks for your answer @damonmaria, I read your post too.
So all I have to do is to install python3-certbot-apache and run sudo certbot renew --force-renewal --preferred-chain="ISRG Root X1" right?

damonmaria · October 13, 2021, 7:28pm

Unfortunately python3-certbot (at least on Ubuntu 18.04 which I was on) is several versions behind and does not support that parameter. I had to install the the snap version of certbot. But then yes, that parameter will do it. Obviously update whichever ‘cron’ or similar process you use to renew as well.

Neeloj · October 14, 2021, 6:14am

very usefull informations I have also 18.04 so if that parameter doesnt support I have to install/upgrade it.

thank you so much @damonmaria