Help with SSL certs and keys for foreman, katello, and remote_execution

I need some help with SSL certs and keys.

When running Remote Execution jobs, they run, but eventually reports failed in Tasks.

foreman-1.22.2-1.el7.noarch
tfm-rubygem-foreman_remote_execution-1.8.2-1.fm1_22.el7.noarch
katello-3.12.3-1.el7.noarch

One RHEL7 server runs all pieces, katello, foreman, proxy, etc.
This was originally built using using katello 3.4 (foreman-installer --scenario katello), with some upgrades along the way. Problem started after upgrade to katello 3.12 and foreman 1.22. There may be some puppet files installed, but we do not make use of Puppet at all. We are working with ansible.

I think something needs updated with my certs in /etc/foreman-proxy/settings.yml and/or /etc/smart_proxy_dynflow_core/settings.yml.

I have found these files on my server:
under /etc/pki/katello-certs-tools/certs
java-client.crt
kat7.xxx.xxx.com-apache.crt
kat7.xxx.xxx.com-foreman-client.crt
kat7.xxx.xxx.com-foreman-proxy-client.crt
kat7.xxx.xxx.com-foreman-proxy.crt
kat7.xxx.xxx.com-puppet-client.crt
kat7.xxx.xxx.com-qpid-broker.crt
kat7.xxx.xxx.com-qpid-client-cert.crt
kat7.xxx.xxx.com-qpid-router-client.crt
kat7.xxx.xxx.com-qpid-router-server.crt
kat7.xxx.xxx.com-tomcat.crt
katello-default-ca.crt
katello-server-ca.crt
pulp-client.crt

under /etc/pki/katello-certs-tools/private
java-client.key
kat7.xxx.xxx.com-apache.key
kat7.xxx.xxx.com-foreman-client.key
kat7.xxx.xxx.com-foreman-proxy-client.key
kat7.xxx.xxx.com-foreman-proxy.key
kat7.xxx.xxx.com-puppet-client.key
kat7.xxx.xxx.com-qpid-broker.key
kat7.xxx.xxx.com-qpid-client-cert.key
kat7.xxx.xxx.com-qpid-router-client.key
kat7.xxx.xxx.com-qpid-router-server.key
kat7.xxx.xxx.com-tomcat.key
pulp-client.key

Can someone help me understand which certs and keys should be used in the different settings files?

Hi,

could you provides the output of egrep 'ssl|port' /etc/foreman-proxy/settings.yml | grep -v '^#'?
It should look something like this:

:ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
:ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
:ssl_private_key: /etc/foreman-proxy/ssl_key.pem
:foreman_ssl_ca: /etc/foreman-proxy/foreman_ssl_ca.pem
:foreman_ssl_cert: /etc/foreman-proxy/foreman_ssl_cert.pem
:foreman_ssl_key: /etc/foreman-proxy/foreman_ssl_key.pem
:https_port: 9090
:http_port: 8000

This is from our 1.22/3.12 testing instance.
The corresponding settings in in /etc/smart_proxy_dynflow_core/settings.yml look the same on our installation (except for the port of course).

:ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
:ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
:ssl_private_key: /etc/foreman-proxy/ssl_key.pem
:https_port: 9090
:http_port: 8000

#:foreman_ssl_ca: /etc/foreman-proxy/foreman_ssl_ca.pem
#:foreman_ssl_cert: /etc/foreman-proxy/foreman_ssl_cert.pem
#:foreman_ssl_key: /etc/foreman-proxy/foreman_ssl_key.pem

No idea why this should break during an update, but try de-commenting those foreman_ssl lines and restart foreman-proxy. Afaik, those options need to be set.

Hi @jmskone

Did the solution @areyus suggest work for you?

:database:

:foreman_url: https://kat7.xxx.xxx.com

:foreman_ssl_ca: /etc/foreman-proxy/ssl_ca.pem
:foreman_ssl_cert: /etc/foreman-proxy/ssl_cert.pem
:foreman_ssl_key: /etc/foreman-proxy/ssl_key.pem

:console_auth: false

:listen: 170.4.xx.xxx
:port: 8008

:ssl_ca_file: ssl/ca.pem
:ssl_private_key: ssl/localhost.pem
:ssl_certificate: ssl/certs/localhost.pem

These settings throw an error that the proxy is not reachable.

I’ll assume you are talking about /etc/smart_proxy_dynflow_core/settings.yml, since that’s the only config I could find containing those settings.
The SSL settings look messed up, here is what we have in our environment:

:database: 
:console_auth: true

# URL of the foreman, used for reporting back
:foreman_url: https://foreman.example.com

# SSL settings for client authentication against foreman.
:foreman_ssl_ca: /etc/foreman-proxy/foreman_ssl_ca.pem
:foreman_ssl_cert: /etc/foreman-proxy/foreman_ssl_cert.pem
:foreman_ssl_key: /etc/foreman-proxy/foreman_ssl_key.pem

# Listen on address
:listen: 0.0.0.0

# Listen on port
:port: 8008

:use_https: true
:ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
:ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
:ssl_private_key: /etc/foreman-proxy/ssl_key.pem

In the future, I would ask you to

  • add an info which configs you are talking about when pasting them
  • Include log messages when possible instead of general explanations of the error message, adding which log they appeared in
  • be a bit more verbose in general about whether you tried things out and whether they worked, regardless of suggested things from other community members or things you came up with yourself

This helps us immensely understanding your problem and figuring out how we can help you.

When I try to run with the settings you sent, I get an OpenSSL error in smart_proxy_dynflow_core.log.

/usr/bin/smart_proxy_dynflow_core:23:in `'SSL_accept returned=1 errno=0 state=SSLv2/v3 read client hello
A: http request (OpenSSL::SSL::SSLError)

When I run in the semi-broken state where the job actually runs, then reports as failed, I think it might be running as http.

170.4.83.147 - - [16/Dec/2019:13:51:57 EST] “POST /tasks/status? HTTP/1.1” 200 76
170.4.83.147 - - [16/Dec/2019:13:51:57 EST] “GET /tasks/449227e2-b7ac-444b-9399-6e35283b19a6/status?
HTTP/1.1” 200 8311
170.4.83.147 - - [16/Dec/2019:13:55:32 EST] “GET /tasks/count?state=running HTTP/1.1” 200 29
170.4.83.147 - - [16/Dec/2019:13:55:32 EST] “POST /tasks/launch? HTTP/1.1” 200 110
SSL_connect returned=1 errno=0 state=error: sslv3 alert unsupported certificate (OpenSSL::SSL::SSLEr
ror)
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in connect_nonblock' /opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:inssl_socket_connect’
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:985:in connect' /opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:920:indo_start’
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:909:in start' . . /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_po ol_executor.rb:319:incatch’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/concurrent-ruby-1.1.4/lib/concurrent/executor/ruby_thread_po
ol_executor.rb:319:in block in create_worker'170.4.83.147 - - [16/Dec/2019:14:05:32 EST] "POST /tasks/status? HTTP/1.1" 200 76 170.4.83.147 - - [16/Dec/2019:14:05:32 EST] "GET /tasks/3d592533-7540-483c-a275-f15de9759f4f/status? HTTP/1.1" 200 8311 SIGTERM (SignalException) /opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:170:inselect’
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:170:in block in start' /opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:32:instart’
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:157:in start' /opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/handler/webrick.rb:34:inrun’
/opt/theforeman/tfm-ror52/root/usr/share/gems/gems/rack-2.0.6/lib/rack/server.rb:297:in start' /opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow_core-0.2.2/lib/smart_proxy_dynflow_core/launch er.rb:15:instart’
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow_core-0.2.2/lib/smart_proxy_dynflow_core/launch
er.rb:8:in launch!' /opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow_core-0.2.2/bin/smart_proxy_dynflow_core:32:in<top (required)>’
/usr/bin/smart_proxy_dynflow_core:23:in load' /usr/bin/smart_proxy_dynflow_core:23:in

Interesting. In the second log, there is an OpenSSL error, too. Looks like in that case, the proxy does not recognize Foreman’s SSL certs.
You cound try taking a look at /etc/foreman/settings.yaml and checking which SSL certificate files are configured there. Then take the semi-broken config (your original one) and try to set those files for the foreman_ssl_* settings in /etc/smart_proxy_dynflow_core/settings.yml and /etc/foreman-proxy/settings.yml.
You could also try rerunning foreman-installer to see if that helps. In theory, it should set the values correctly and since being idempotent, should not change anything that is “correct”. I would recommend running “foreman-installer -v --noop” first, though, to see what would be changed without the risk of actual undesired changes.
If nothing else helps, your certificates might have been corrupted. I have heard some rare reports about that happening the last weeks. In that case, you might need to regenerate them, but I cannot provided any steps for that from memory.

Thank you so much, that is what I needed. I ran

  1. foreman-installer and this fixed /etc/foreman-proxy/settings.yml
  2. foreman-installer --enable-foreman-plugin-remote-execution --enable-foreman-proxy-plugin-remote-execution-ssh and this fixed /etc/smart_proxy_dynflow_core/settings.yml

/etc/foreman-proxy/settings.yml now has
:ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
:ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
:ssl_private_key: /etc/foreman-proxy/ssl_:foreman_ssl_ca: /etc/foreman-proxy/foreman_ssl_ca.pem
:foreman_ssl_cert: /etc/foreman-proxy/foreman_ssl_cert.pem
:foreman_ssl_key: /etc/foreman-proxy/foreman_ssl_key.pem

/etc/smart_proxy_dynflow_core/settings.yml now has
:foreman_ssl_ca: /etc/foreman-proxy/foreman_ssl_ca.pem
:foreman_ssl_cert: /etc/foreman-proxy/foreman_ssl_cert.pem
:foreman_ssl_key: /etc/foreman-proxy/foreman_ssl_key.pem
:listen: 0.0.0.0
:port: 8008
:use_https: true
:ssl_ca_file: /etc/foreman-proxy/ssl_ca.pem
:ssl_certificate: /etc/foreman-proxy/ssl_cert.pem
:ssl_private_key: /etc/foreman-proxy/ssl_key.pem

Glad to hear that fixed your issue :slight_smile:
If everything works as expected now, pleas mark the thread as solved so others can see that a solution was found.

Regards