The only applicable proxy is down, unknown protocol

Support
1

Problem:

Foreman proxy is down. I am not sure why. Instance is over 6 months old.

The foreman-maintain service status are all active. Issue appears to be related to either certificate/SSL misconfiguration, but I have not found anything overwhelmingly obvious that i can fix.

I belive the key line in error message is:

<OpenSSL::SSL::SSLError>: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol

Not any recent upgrades. Not any recent system changes.

I get the following error on most, if not all, jobs including “Scheduled Jobs” and hammer cli.

Failed to initialize: RuntimeError - The only applicable proxy foremanURL is down

Oddly enough, the Infustructure > Smart Proxies page shows that Communication Status is green.

I have seen a few similar issues on this forum, but those solutions did not help me out.

Expected outcome:
Proxy should be working.

Foreman and Proxy versions:
2.4.0

Foreman and Proxy plugin versions:

|foreman-tasks 4.0.1
|foreman_remote_execution 4.3.0
|katello 4.0.1.1

Distribution and version:
Centos 7

Other relevant data:
Here is the full proxy output:

Error processing request '957eb5e4-2bba-466e-9808-2ff10d28313e: <OpenSSL::SSL::SSLError>: SSL_connect returned=1 errno=0 state=SSLv2/v3 read server hello A: unknown protocol
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in `connect_nonblock'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/protocol.rb:44:in `ssl_socket_connect'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:985:in `connect'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:920:in `do_start'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:909:in `start'
/opt/rh/rh-ruby25/root/usr/share/ruby/net/http.rb:1458:in `request'
/usr/share/foreman-proxy/lib/proxy/request.rb:48:in `send_request'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.3.0/lib/smart_proxy_dynflow/callback.rb:24:in `relay'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.3.0/lib/smart_proxy_dynflow/callback.rb:30:in `relay'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.3.0/lib/smart_proxy_dynflow/helpers.rb:5:in `relay_request'
/opt/theforeman/tfm/root/usr/share/gems/gems/smart_proxy_dynflow-0.3.0/lib/smart_proxy_dynflow/api.rb:62:in `block in <class:Api>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/opt/theforeman/tfm/root/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/opt/rh/rh-ruby25/root/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Screenshots:

image
image1943×420 61.6 KB

2

Can you run the following command on the foreman host:

foreman-maintain health check

How many smart proxies you have ?
Do you use custom certificates or autosigned ?

You need first to be sure that foreman host is up and running.

Maybe a foreman-installer (backup your system before) can fix it quickly.

From the screenshot, the first message is weird, but the others can be ignored.

3

Here is that output:
# foreman-maintain health check

Running ForemanMaintain::Scenario::FilteredScenario
================================================================================
Check number of fact names in database:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running:                               [OK]
--------------------------------------------------------------------------------
Check whether all services are running using the ping call:           [OK]
--------------------------------------------------------------------------------
Check for paused tasks:                                               [OK]
--------------------------------------------------------------------------------
Check to verify no empty CA cert requests exist:                      [OK]
--------------------------------------------------------------------------------

Using a single box deployment, so 1 proxy.

Yes, I do have custom certs for Katello that are deployed via foreman-installer and haven’t had any issues the last 6+ months.

I am concerned the issue is related to that SSL V2/V3 error that says unknown protocol. No idea why SSL v2/v3 is anywhere in the application, and haven’t found anything helpful yet. Note, on the web-browser, when I inspect my cert, its says it is secured via TLS 1.2, so that end of things appear OK.

Ill try playing around with foreman-installer. Maybe that will reinitialize the proxy back into a working state.

4

@kobena, your suggestion worked! Looks like i’m working again now:

  • took a snapshot
  • yum update -y
  • foreman-maintain service stop
  • foreman-installer
    • after completion: proxy working!
  • reboot
  • still working!
5

Hi @barn

Excellent news, glad to have been able to help you.

You can close this topic if you consider your problem solved.