Problem:
When attempting to run hammer host reboot --id 3 the below error occurs: ERF12-1772 [ProxyAPI::ProxyException]: Unable to perform power BMC operation ([RestClient::BadRequest]: 400 Bad Request) for proxy https://<proxyurl>:8443/bmc
Expected outcome:
The host with id 3 reboots. Foreman and Proxy versions:
Foreman proxy: 3.1.1 Foreman and Proxy plugin versions:
BMC: 3.1.1 Distribution and version:
RHEL 8.5 Other relevant data:
Note that the BMC tab in the web interface shows the same error.
What are the troubleshooting steps for this error?
Hey, assuming you have a BMC interface set with an IP username and password. Then you also need to have a BMC proxy running on that network (foreman-proxy process with BMC module turned on) and you see such proxy in Administer - Smart Proxies (use refresh feature to see the BMC feature). Finally, you need to ead over to Subnet in Foreman and associate the BMC Proxy with your subnet.
If you have issues, investigate in Foreman and Smart Proxy logs. Enable debug logging and pastebin errors with stacktraces here.
The many error messages are mostly: Error while monitoring: getaddrinfo: Name or service not known. I am not sure if this is related. I did check that the hostname was set on the system, but there might be a Foreman specific hostname configuration for the smart proxy that I am unaware of.
Upon checking the logs for the BMC issue, I found the following in /var/log/foreman/production.log:
2022-03-14T11:29:57 [I|app|57908bec] Backtrace for 'Failed to fetch bmc information' error (ProxyAPI::ProxyException): ERF12-1772 [ProxyAPI::ProxyException]: Unable to perform power BMC operation ([RestClient::BadRequest]: 400 Bad Request) for proxy https://<foremanhostname>:8443/bmc
57908bec | /usr/share/foreman/lib/proxy_api/bmc.rb:64:in `rescue in power'
57908bec | /usr/share/foreman/lib/proxy_api/bmc.rb:47:in `power'
[...long stacktrace clipped...]
57908bec | /usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
57908bec | /usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
57908bec | /usr/share/gems/gems/puma-5.5.0/lib/puma/configuration.rb:249:in `call'
57908bec | /usr/share/gems/gems/puma-5.5.0/lib/puma/request.rb:77:in `block in handle_request'
57908bec | /usr/share/gems/gems/puma-5.5.0/lib/puma/thread_pool.rb:340:in `with_force_shutdown'
57908bec | /usr/share/gems/gems/puma-5.5.0/lib/puma/request.rb:76:in `handle_request'
57908bec | /usr/share/gems/gems/puma-5.5.0/lib/puma/server.rb:447:in `process_client'
57908bec | /usr/share/gems/gems/puma-5.5.0/lib/puma/thread_pool.rb:147:in `block in spawn_thread'
I also found the following in /var/log/foreman-proxy/proxy.log:
022-03-14T11:29:54 dfc4623a [W] SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:
`Excon.defaults[:ssl_ca_path] = path_to_certs`
`ENV['SSL_CERT_DIR'] = path_to_certs`
`Excon.defaults[:ssl_ca_file] = path_to_file`
`ENV['SSL_CERT_FILE'] = path_to_file`
`Excon.defaults[:ssl_verify_callback] = callback`
(see OpenSSL::SSL::SSLContext#verify_callback)
or:
`Excon.defaults[:ssl_verify_peer] = false` (less secure).
: <Excon::Error::Certificate>: SSL_connect returned=1 errno=0 state=error: certificate verify failed (self signed certificate) (OpenSSL::SSL::SSLError) Unable to verify certificate. This may be an issue with the remote host or with Excon. Excon has certificates bundled, but these can be customized:
`Excon.defaults[:ssl_ca_path] = path_to_certs`
`ENV['SSL_CERT_DIR'] = path_to_certs`
`Excon.defaults[:ssl_ca_file] = path_to_file`
`ENV['SSL_CERT_FILE'] = path_to_file`
`Excon.defaults[:ssl_verify_callback] = callback`
(see OpenSSL::SSL::SSLContext#verify_callback)
or:
`Excon.defaults[:ssl_verify_peer] = false` (less secure).
/usr/share/gems/gems/excon-0.76.0/lib/excon/ssl_socket.rb:131:in `connect_nonblock'
/usr/share/gems/gems/excon-0.76.0/lib/excon/ssl_socket.rb:131:in `initialize'
/usr/share/gems/gems/excon-0.76.0/lib/excon/connection.rb:465:in `new'
[...clipped...]
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
It seems that there is an SSL error, this makes sense since the BMC has a self-signed certificate.
How do I instruct Foreman to ignore certificate errors?
The proxy log explains it nicely, you are connecting to HTTPS Redfish endpoint and your operating system (smart proxy OS) does not have CA certificate in its database, thus it fails to connect. This is expected behavior.
There is a redfish_verify_ssl smart proxy setting in bmc.yml:
# Redfish provider HTTPS certificate verification:
# - If your BMCs just have the vendor-supplied self-signed certificates, you can set
# bmc_redfish_verify_ssl to false. The default is to perform certificate verification.
#:redfish_verify_ssl: true
Flip that to false if that’s what you want to do, or download the system CA to the smart-proxy.
I do not have that option in /etc/foreman-proxy/settings.d/bmc.yml. I added it, set it to false, and restarted, but I still have the same errors as before. Is my version behind or am I missing something else?
I edited /usr/share/gems/gems/redfish_client-0.5.2/lib/redfish_client/connector.rb and set ssl_verify_peer: false in the initialize function. This is obviously not the best solution, but it does confirm where the issue is.
Upon further investigation I have found that if I comment out line 6 from /usr/share/foreman-proxy/modules/bmc/bmc_plugin.rb, the value specified in bmc.yml is respected.
That line reads: validate :redfish_verify_ssl, :boolean => true
This appears to be a bug, but I am not experienced enough in Ruby or Foreman to be sure.
I’ve run into this as well. In my environment, we have not been installing a custom x509 cert onto bmcs. Even if we were, tls validation being stuck on still presents a boot-strapping problem.
