Foreman 1.15.6 mysql TLS 1.2

maxpj · April 17, 2020, 5:52pm

Here is our setup ( we will upgrade soon but for now that’s what we’re using ) :

Foreman 1.15.6 running inside Kubernetes AWS EKS 1.15

Our database is in AWS I just upgraded the database from mysql5.6 to mysql 5.7 and as soon as I turn TLS1.2 on on the database foreman does not work anymore. It seems to work just fine when I add the perameter in the db to accept TLS1.1also. As soon as I remove 1.1 from the authorized TLS version everything stops working

I did change the foreman and puppet config to add SSL_PROTOCOL TLS1.2 but it did not change anything.

Any ideas what I could be missing here?

Thanks

tbrisker · April 19, 2020, 9:50am

What is the error you are seeing in production.log?
Keep in mind that running Foreman inside Kubernetes with external mysql db on aws is not a supported or tested environment - so you will likely be on your own to resolve this issue. Also chances of someone having such an old version still running that can help with debugging are slim.
Foreman 2.0 dropped support for MySQL, so while planning your upgrade you should also take into consideration the need to migrate to PostgreSQL - there are instructions in the manual for what needs to be done.

maxpj · April 20, 2020, 1:00pm

This is the error:

Mysql2::Error: SSL connection error: unknown error number

That being said, it works when I accept TLS1.1 & TLS1.2 and as soon as I remove the TLS1.1 from the accepted connection on the database it stops working.

And yeah I saw that mysql was no longer supported in 2.0 and we will be working on it soon.

tbrisker · April 20, 2020, 1:05pm

Can you share the full stack trace? it’s possible that the version of the mysql adapter in 1.15 is so old that it doesn’t support tls1.2 connections, or that some other library doesn’t support it. You can also try initializing an ssl connection from the foreman server to the db with openssl in debug mode to see if the connection succeeds or not.

maxpj · April 20, 2020, 2:41pm

It’s installing version 0.49 of mysql2 adapter.

Enabling site 05-foreman-ssl.
To activate the new configuration, you need to run:
  service apache2 reload
Considering dependency setenvif for ssl:
Module setenvif already enabled
Considering dependency mime for ssl:
Module mime already enabled
Considering dependency socache_shmcb for ssl:
Enabling module socache_shmcb.
Enabling module ssl.
See /usr/share/doc/apache2/README.Debian.gz on how to configure SSL and create self-signed certificates.
To activate the new configuration, you need to run:
  service apache2 restart
rake aborted!
Mysql2::Error: SSL connection error: unknown error number
/usr/share/foreman/vendor/ruby/2.3.0/gems/mysql2-0.4.9/lib/mysql2/client.rb:89:in `connect'
/usr/share/foreman/vendor/ruby/2.3.0/gems/mysql2-0.4.9/lib/mysql2/client.rb:89:in `initialize'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `new'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `mysql2_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:in `new_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:in `checkout_new_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:in `acquire_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in `block in checkout'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_handling.rb:87:in `connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/migration.rb:916:in `initialize'
/usr/share/foreman/app/services/foreman/plugin.rb:280:in `new'
/usr/share/foreman/app/services/foreman/plugin.rb:280:in `pending_migrations'
/usr/share/foreman/app/services/foreman/plugin.rb:235:in `permission'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_setup-5.0.0/lib/foreman_setup/engine.rb:27:in `block (3 levels) in <class:Engine>'
/usr/share/foreman/app/services/foreman/plugin.rb:219:in `instance_eval'
/usr/share/foreman/app/services/foreman/plugin.rb:219:in `security_block'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_setup-5.0.0/lib/foreman_setup/engine.rb:26:in `block (2 levels) in <class:Engine>'
/usr/share/foreman/app/services/foreman/plugin.rb:66:in `instance_eval'
/usr/share/foreman/app/services/foreman/plugin.rb:66:in `register'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_setup-5.0.0/lib/foreman_setup/engine.rb:18:in `block in <class:Engine>'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:30:in `instance_exec'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:30:in `run'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:55:in `block in run_initializers'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:54:in `run_initializers'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/application.rb:352:in `initialize!'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/railtie.rb:194:in `public_send'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/railtie.rb:194:in `method_missing'
/usr/share/foreman/config/environment.rb:5:in `<top (required)>'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `block in require'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:240:in `load_dependency'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/application.rb:328:in `require_environment!'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/application.rb:457:in `block in run_tasks_blocks'
Tasks: TOP => apipie:cache => environment
(See full trace by running task with --trace)
Apipie cache enabled but not present yet. Run apipie:cache rake task to speed up API calls.
rake aborted!
Mysql2::Error: SSL connection error: unknown error number
/usr/share/foreman/vendor/ruby/2.3.0/gems/mysql2-0.4.9/lib/mysql2/client.rb:89:in `connect'
/usr/share/foreman/vendor/ruby/2.3.0/gems/mysql2-0.4.9/lib/mysql2/client.rb:89:in `initialize'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `new'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/mysql2_adapter.rb:18:in `mysql2_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:438:in `new_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:448:in `checkout_new_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:422:in `acquire_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:349:in `block in checkout'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:348:in `checkout'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:263:in `block in connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:262:in `connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract/connection_pool.rb:571:in `retrieve_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_handling.rb:113:in `retrieve_connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/connection_handling.rb:87:in `connection'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activerecord-4.2.8/lib/active_record/migration.rb:916:in `initialize'
/usr/share/foreman/app/services/foreman/plugin.rb:280:in `new'
/usr/share/foreman/app/services/foreman/plugin.rb:280:in `pending_migrations'
/usr/share/foreman/app/services/foreman/plugin.rb:235:in `permission'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_setup-5.0.0/lib/foreman_setup/engine.rb:27:in `block (3 levels) in <class:Engine>'
/usr/share/foreman/app/services/foreman/plugin.rb:219:in `instance_eval'
/usr/share/foreman/app/services/foreman/plugin.rb:219:in `security_block'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_setup-5.0.0/lib/foreman_setup/engine.rb:26:in `block (2 levels) in <class:Engine>'
/usr/share/foreman/app/services/foreman/plugin.rb:66:in `instance_eval'
/usr/share/foreman/app/services/foreman/plugin.rb:66:in `register'
/usr/share/foreman/vendor/ruby/2.3.0/gems/foreman_setup-5.0.0/lib/foreman_setup/engine.rb:18:in `block in <class:Engine>'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:30:in `instance_exec'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:30:in `run'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:55:in `block in run_initializers'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/initializable.rb:54:in `run_initializers'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/application.rb:352:in `initialize!'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/railtie.rb:194:in `public_send'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/railtie.rb:194:in `method_missing'
/usr/share/foreman/config/environment.rb:5:in `<top (required)>'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `block in require'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:240:in `load_dependency'
/usr/share/foreman/vendor/ruby/2.3.0/gems/activesupport-4.2.8/lib/active_support/dependencies.rb:274:in `require'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/application.rb:328:in `require_environment!'
/usr/share/foreman/vendor/ruby/2.3.0/gems/railties-4.2.8/lib/rails/application.rb:457:in `block in run_tasks_blocks'
Tasks: TOP => db:migrate => environment
(See full trace by running task with --trace)
Apipie cache enabled but not present yet. Run apipie:cache rake task to speed up API calls.

tbrisker · April 20, 2020, 3:39pm

I’m afraid you’re likely on your own debugging this issue, this isn’t a supported setup and this could be caused by many different factors. We don’t do any testing of running foreman inside a container or connecting to external cloud provider databases, and the version you are using is so old that it is hard to even attempt to debug it blindly. If you do figure it out pleas share the outcome in case anyone runs into a similar issue in the future.

maxpj · April 20, 2020, 3:51pm

Just here to report back that in the end we rebuilded the docker image and therefore updated the following lib and everything works now! cheers

libmysqlclient-dev

Thanks for your help