Hammer -- SSL certificate verification failed

Stefan_Lasiewski · February 4, 2021, 3:03am

Problem:

Curl can connect to my Foreman webserver just fine:

$ curl --head https://foreman.example.org
HTTP/1.1 302 Found
Date: Thu, 04 Feb 2021 02:52:05 GMT
...

But Hammer cannot:

$ hammer ping 
SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/ssl/certs/intermediate-cert.pem

Make sure the location contains an unexpired and valid CA certificate for https://foreman.example.org.
$

I suspect that something on the server is misconfigured. Foreman itself seems to work okay as far as I can tell.

How can I verify that the Foreman URL works with the ca file specified by ssl_ca_file?

Expected outcome:

I expected Hammer to use the server certificate and proceed, just like curl.

Foreman and Proxy versions:

Foreman: 1.24.3
Foreman Proxy: 1.24.3

Foreman and Proxy plugin versions:

Foreman Discovery: 16.0.1

Distribution and version:

Ubuntu 18.04.5 LTS

Other relevant data:

Nothing in the server logs.

$ less .hammer/log/hammer.log 
[ERROR 2021-02-03T19:00:44 API] SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get issuer certificate)
[ERROR 2021-02-03T19:00:44 Exception] SSL certificate verification failed
Make sure you configured the correct URL and have the server's CA certificate installed on your system.

The following configuration option were used for the SSL connection:
  ssl_ca_file = /etc/ssl/certs/intermediate-cert.pem

Make sure the location contains an unexpired and valid CA certificate for https://foreman.example.org.
[ERROR 2021-02-03T19:00:44 Exception] 

RestClient::SSLCertificateNotVerified (SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get issuer certificate)):
    /usr/lib/ruby/vendor_ruby/restclient/request.rb:758:in `rescue in transmit'
    /usr/lib/ruby/vendor_ruby/restclient/request.rb:642:in `transmit'
    /usr/lib/ruby/vendor_ruby/restclient/request.rb:145:in `execute'
    /usr/lib/ruby/vendor_ruby/restclient/request.rb:52:in `execute'
    /usr/lib/ruby/vendor_ruby/restclient/resource.rb:51:in `get'
    /usr/lib/ruby/vendor_ruby/apipie_bindings/api.rb:327:in `call_client'
    /usr/lib/ruby/vendor_ruby/apipie_bindings/api.rb:240:in `http_call'
    /usr/lib/ruby/vendor_ruby/apipie_bindings/api.rb:190:in `call_action'
    /usr/lib/ruby/vendor_ruby/apipie_bindings/api.rb:185:in `call'
    /usr/lib/ruby/vendor_ruby/apipie_bindings/resource.rb:21:in `call'
    /usr/lib/ruby/vendor_ruby/hammer_cli/apipie/command.rb:53:in `send_request'
    /usr/lib/ruby/vendor_ruby/hammer_cli_foreman/commands.rb:188:in `send_request'
    /usr/lib/ruby/vendor_ruby/hammer_cli/apipie/command.rb:34:in `execute'
    /usr/lib/ruby/vendor_ruby/clamp/command.rb:63:in `run'
    /usr/lib/ruby/vendor_ruby/hammer_cli/abstract.rb:76:in `run'
    /usr/lib/ruby/vendor_ruby/clamp/subcommand/execution.rb:11:in `execute'
    /usr/lib/ruby/vendor_ruby/clamp/command.rb:63:in `run'
    /usr/lib/ruby/vendor_ruby/hammer_cli/abstract.rb:76:in `run'
    /usr/lib/ruby/vendor_ruby/clamp/subcommand/execution.rb:11:in `execute'
    /usr/lib/ruby/vendor_ruby/clamp/command.rb:63:in `run'
    /usr/lib/ruby/vendor_ruby/hammer_cli/abstract.rb:76:in `run'
    /usr/lib/ruby/vendor_ruby/clamp/command.rb:132:in `run'
    /usr/bin/hammer:147:in `<main>'

Stefan_Lasiewski · February 4, 2021, 3:17am

Unsure if this is related, but earlier today I installed the Templates Proxy because I have a multi-homed system. I can’t get those certs to work either /var/log/foreman-proxy/proxy.log gives errors like this:

2021-02-03T19:14:07 3c731976 [I] Started GET /unattended/finish 
2021-02-03T19:14:07 3c731976 [E] Failed to proxy /finish for {"splat"=>[], "captures"=>["finish"], "kind"=>"finish"}: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable
 to get local issuer certificate)
2021-02-03T19:14:07 3c731976 [W] Failed to proxy /finish for {"splat"=>[], "captures"=>["finish"], "kind"=>"finish"}: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable
 to get local issuer certificate)
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

I’m unsure if my server is misconfigured or not.

Stefan_Lasiewski · February 5, 2021, 12:27am

I’ve traced the file down to the /etc/hammer/cli.modules.d/foreman.yml file.

I don’t understand why foreman-installer is adding the ssl_ca_file here. There doesn’t seem to be an option to do anything else. Using ssl_ca_dirs=/etc/ssl/certs/ would also work, but there is no option for that.

$ cat /etc/hammer/cli.modules.d/foreman.yml

:foreman:
  # Enable/disable foreman commands
  :enable_module: true

  # Your foreman server address
  :host: 'https://foreman.example.org'

:ssl:
  :ssl_ca_file: '/etc/ssl/certs/intermediate-cert.pem'

If I comment out the ssl_ca_file it works:

$ cat /etc/hammer/cli.modules.d/foreman.yml
:foreman:
  # Enable/disable foreman commands
  :enable_module: true

  # Your foreman server address
  :host: 'https://foreman.example.org'

:ssl:
  #:ssl_ca_file: '/etc/ssl/certs/intermediate-cert.pem'
$ hammer ping
database: 
    Status:          ok
    Server Response: Duration: 0ms

$

Stefan_Lasiewski · February 5, 2021, 2:57am

Setting a directory also works:

$ hammer --ssl-ca-path /etc/ssl/certs ping
database: 
    Status:          ok
    Server Response: Duration: 0ms
$

However, there is no option to configure this with foreman-installer. foreman-answers.yaml has this:

foreman::cli:
  foreman_url:
  version: installed
  manage_root_config: true
  username:
  password:
  refresh_cache: false
  request_timeout: 120
  ssl_ca_file:
  hammer_plugin_prefix: ruby-hammer-cli-

SilverWizard · December 22, 2021, 2:04pm

Hi Stefan,

Did you ever find a solution ?
I have seen this same error in version 2.5 while running “foreman-maintain health check”.

However while I’m not encountering any issues in functionality at the moment, I would like to solve this issue.
“hammer ping” command executes without any issue on my end but the healthcheck still reports the issue.

In /etc/hammer/cli.modules.d/foreman.yml

I’ve set the proper FQDN, ssl is enabled and a certificate path is set to the proper CA cert.

purecopper · August 22, 2022, 2:31am

Hi.

I stumbled upon this old thread when I experienced a similar verification failure message today after running a foreman-maintain update. I’m currently running Foreman-Katello 3.3.0.

I thought I should go ahead and post what I found to resolve it in my instance in case others have a similar experience updating…

The cause in my case was the permissions on the certificate file listed in the message had changed during the update; the certificate was no longer world readable.

The resolution was a simple chmod o+r .