Certificates are invalid after update to 2.4.0 and Katello 4.0

Support
, ,
1

Problem:
After solving the issue with puppet (Update to 2.4.0 with Katello 4.0 fails - #6 by jjeffers), now “katello:clean_backend_objects” fails:

=============================================
Upgrade Step 3/3: katello:clean_backend_objects. This may take a long while.
Failed upgrade task: katello:clean_backend_objects, see logs for more information.

With “foreman-rake katello:clean_backend_objects --trace” I get this output:

foreman-rake katello:clean_backend_objects --trace
** Invoke katello:clean_backend_objects (first_time)
** Invoke environment (first_time)
** Execute environment
** Invoke katello:check_ping (first_time)
** Invoke environment
** Invoke dynflow:client (first_time)
** Invoke environment
** Execute dynflow:client
** Execute katello:check_ping
** Execute katello:clean_backend_objects
The following changes will not actually be performed. Rerun with COMMIT=true to apply the changes
rake aborted!
RestClient::SSLCertificateNotVerified: SSL_connect returned=1 errno=0 state=error: certificate verify failed (error number 1)
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:758:in rescue in transmit' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:642:in transmit’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in get' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:94:in get_response’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:74:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:28:in retrieve’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:35:in retrieve_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:16:in populate!’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:106:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in mon_synchronize’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in standard_exception_handling’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in load' /opt/rh/rh-ruby25/root/usr/bin/rake:23:in ’

Caused by:
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (error number 1)
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in connect_nonblock' /opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in ssl_socket_connect’
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:985:in connect' /opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:920:in do_start’
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:909:in start' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:715:in transmit’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in execute’
/opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:51:in get' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:94:in get_response’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/base.rb:74:in call' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:28:in retrieve’
/opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.13.1/lib/runcible/resources/consumer.rb:35:in retrieve_all' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:16:in populate!’
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-4.0.0/lib/katello/tasks/clean_backend_objects.rake:106:in block (2 levels) in <top (required)>' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in block in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:251:in execute’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:195:in block in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/ruby/monitor.rb:226:in mon_synchronize’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:188:in invoke_with_call_chain' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/task.rb:181:in invoke’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:160:in invoke_task' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block (2 levels) in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in each' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:116:in block in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:125:in run_with_threads' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:110:in top_level’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:83:in block in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:186:in standard_exception_handling’
/opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/lib/rake/application.rb:80:in run' /opt/rh/rh-ruby25/root/usr/share/gems/gems/rake-12.3.0/exe/rake:27:in <top (required)>’
/opt/rh/rh-ruby25/root/usr/bin/rake:23:in load' /opt/rh/rh-ruby25/root/usr/bin/rake:23:in ’
Tasks: TOP => katello:clean_backend_objects

Used katello-certs-check on various certificates and got problems with these sets:

/etc/foreman-proxy/foreman_ssl_ca.pem
/etc/foreman-proxy/foreman_ssl_key.pem
/etc/foreman-proxy/foreman_ssl_cert.pem

/etc/pki/katello/puppet/puppet_client_ca.crt
/etc/pki/katello/puppet/puppet_client.key
/etc/pki/katello/puppet/puppet_client.crt

/etc/foreman/proxy_ca.pem
/etc/foreman/client_key.pem
/etc/foreman/client_cert.pem

The check fails at this step:

Checking CA bundle against the certificate file:
[FAIL]

The /etc/pki/katello/puppet/puppet_client_ca.crt does not verify the /etc/pki/katello/puppet/puppet_client.crt
/etc/pki/katello/puppet/puppet_client.crt: C = US, ST = North Carolina, O = FOREMAN, OU = PUPPET, CN = my.system.com
error 26 at 0 depth lookup:unsupported certificate purpose

All x509 v3 extensions are correct, according to the manpage of openssl, but couldn’t
find information about the Netscape extensions.

Because it is a VM I created a snapshot and run the foreman-installer:

foreman-installer --scenario katello --certs-update-server
foreman-installer --scenario katello --certs-update-all

Both times I got the error with “katello:clean_backend_objects” and don’t know
what to do now.
Certificates had been generated by the foreman-installer, not custom certificates had
been used.

Expected outcome:
Installer works after update to 2.4.0 with Katello 4.0.

Foreman and Proxy versions:
2.4.0 and proxies are not updated yet, running on 2.2.3

Foreman and Proxy plugin versions:
foreman-tasks 4.0.1
foreman_ansible 6.2.0
foreman_remote_execution 4.3.0
katello 4.0.0

Distribution and version:
CentOS 7

2

Seems like you may be hitting this issue, which is slated to be fixed in 4.0.1.

https://projects.theforeman.org/issues/32475

2 Likes