Problem:
I have 2 hosts (both Gigabyte) that never have power options after upgrading from 3.0 to 3.2. I think what’s happening is the forman interface is timing out before the BMC responds so the widgets never turn green. Is there a place I can frob the timeout in foreman’s interface to verify the problem?
These two servers take 6-10 seconds to respond and the interface seems to decide there are no power options within 3 seconds. Expected outcome:
Foreman and Proxy versions:
Foreman Version 3.2.1
Proxy 3.2.1 Foreman and Proxy plugin versions:
foreman (3.2.1-2)
foreman-proxy (3.2.1-1) Distribution and version:
Ubuntu 20.04.4 LTS Other relevant data:
</var/log/foreman-proxy/proxy.log>
2022-06-13T14:27:14 b88a4503 [D] Finished DNS query getname for 'X.X.X.X' in 1.56 ms
2022-06-13T14:27:14 b88a4503 [D] Finished DNS query getaddresses for 'foo.example.com' in 1.09 ms
2022-06-13T14:27:14 b88a4503 [D] require_ssl_client_verification: skipping, non-HTTPS request
2022-06-13T14:27:20 b88a4503 [D] No GIGABYTE specific overrides available - using generic Redfish calls
2022-06-13T14:27:20 b88a4503 [I] Finished GET /bmc/x.x.x.x/chassis/power/status with 200 (6907.38 ms)
UPDATE
Just discovered the link to the legacy UI, it works there.
These Gigabyte BMCs are SLOW!
2022-06-13T15:50:22 750628c4 [I] Started GET /bmc/X.X.X.X/chassis/power/status bmc_provider=Redfish
2022-06-13T15:50:22 750628c4 [D] verifying remote client X.X.X.X against trusted_hosts ["foo.example.com", "X.X.X.X", "bar.example.com", "X.X.X.X"]
2022-06-13T15:50:22 750628c4 [D] Finished DNS query getname for 'X.X.X.X' in 1.85 ms
2022-06-13T15:50:22 750628c4 [D] Finished DNS query getaddresses for 'foo.example.com' in 1.22 ms
2022-06-13T15:50:22 750628c4 [D] require_ssl_client_verification: skipping, non-HTTPS request
2022-06-13T15:50:29 750628c4 [D] No GIGABYTE specific overrides available - using generic Redfish calls
2022-06-13T15:50:29 750628c4 [I] Finished GET /bmc/X.X.X.X/chassis/power/status with 200 (7502.45 ms)
2022-06-13T15:50:29 750628c4 [I] Started GET /bmc/X.X.X.X/lan/ip bmc_provider=Redfish
2022-06-13T15:50:29 750628c4 [D] verifying remote client X.X.X.X against trusted_hosts ["foo.example.com", "X.X.X.X", "bar.example.com", "X.X.X.X"]
2022-06-13T15:50:29 750628c4 [D] Finished DNS query getname for 'X.X.X.X' in 1.49 ms
2022-06-13T15:50:29 750628c4 [D] Finished DNS query getaddresses for 'foo.example.com' in 1.08 ms
2022-06-13T15:50:29 750628c4 [D] require_ssl_client_verification: skipping, non-HTTPS request
2022-06-13T15:50:43 750628c4 [D] No GIGABYTE specific overrides available - using generic Redfish calls
2022-06-13T15:50:57 750628c4 [W] BMC has multiple NICs? - using first
2022-06-13T15:50:57 750628c4 [I] Finished GET /bmc/X.X.X.X/lan/ip with 200 (28157.26 ms)
2022-06-13T15:50:57 750628c4 [I] Started GET /bmc/X.X.X.X/lan/netmask bmc_provider=Redfish
2022-06-13T15:50:57 750628c4 [D] verifying remote client X.X.X.X against trusted_hosts ["foo.example.com", "X.X.X.X", "bar.example.com", "X.X.X.X"]
2022-06-13T15:50:57 750628c4 [D] Finished DNS query getname for 'X.X.X.X' in 1.57 ms
2022-06-13T15:50:57 750628c4 [D] Finished DNS query getaddresses for 'foo.example.com' in 1.15 ms
2022-06-13T15:50:57 750628c4 [D] require_ssl_client_verification: skipping, non-HTTPS request
2022-06-13T15:51:06 750628c4 [D] No GIGABYTE specific overrides available - using generic Redfish calls
2022-06-13T15:51:13 750628c4 [W] BMC has multiple NICs? - using first
2022-06-13T15:51:13 750628c4 [I] Finished GET /bmc/X.X.X.X/lan/netmask with 200 (16088.55 ms)
2022-06-13T15:51:13 750628c4 [I] Started GET /bmc/X.X.X.X/lan/mac bmc_provider=Redfish
2022-06-13T15:51:13 750628c4 [D] verifying remote client X.X.X.X against trusted_hosts ["foo.example.com", "X.X.X.X", "bar.example.com", "X.X.X.X"]
2022-06-13T15:51:13 750628c4 [D] Finished DNS query getname for 'X.X.X.X' in 1.49 ms
2022-06-13T15:51:13 750628c4 [D] Finished DNS query getaddresses for 'foo.example.com' in 1.06 ms
2022-06-13T15:51:13 750628c4 [D] require_ssl_client_verification: skipping, non-HTTPS request
2022-06-13T15:51:26 750628c4 [D] No GIGABYTE specific overrides available - using generic Redfish calls
2022-06-13T15:51:34 750628c4 [W] BMC has multiple NICs? - using first
2022-06-13T15:51:34 750628c4 [I] Finished GET /bmc/X.X.X.X/lan/mac with 200 (20692.3 ms)
2022-06-13T15:51:34 750628c4 [I] Started GET /bmc/X.X.X.X/lan/gateway bmc_provider=Redfish
2022-06-13T15:51:34 750628c4 [D] verifying remote client X.X.X.X against trusted_hosts ["foo.example.com", "X.X.X.X", "bar.example.com", "X.X.X.X"]
2022-06-13T15:51:34 750628c4 [D] Finished DNS query getname for 'X.X.X.X' in 1.6 ms
2022-06-13T15:51:34 750628c4 [D] Finished DNS query getaddresses for 'foo.example.com' in 1.05 ms
2022-06-13T15:51:34 750628c4 [D] require_ssl_client_verification: skipping, non-HTTPS request
2022-06-13T15:51:45 750628c4 [D] No GIGABYTE specific overrides available - using generic Redfish calls
2022-06-13T15:51:53 750628c4 [W] BMC has multiple NICs? - using first
2022-06-13T15:51:53 750628c4 [I] Finished GET /bmc/X.X.X.X/lan/gateway with 200 (18959.26 ms)
Correct, the new UI times out before the BMC ever responds to the proxy. The legacy UI will continue to spin until something comes back or it hits a much longer timeout.
Thanks for letting us know, we will take a look into it soon. I didn’t know it takes so long for the BMC to respond…
this could be a great use case for web socket where the frontend gets a push with the BMC response,
though since we don’t use it yet maybe we could look on the BMC status and if it’s still empty, keep requesting until we get new data for it, with reasonable timeout.
Looking at the logs from these Gigabyte servers they can take as long as 28 seconds to pull data depending on how much info you poke them for, must be powered by a C64. Our other systems (supermicro) usually average a second or 2 and they are all on the same network so it has to be the BMC itself. So long as I have access to the legacy interface as a work around it’s not a big deal for me. I mostly just wanted to document this behaviour so others aren’t chasing their tails trying to figure out why it doesn’t seem to work.
I think the issue is the timeout of our frontend when it waits for Foreman to respond. @Ron_Lavi is there something we can do to make the frontend be more patient (up to 30 seconds) in some requests? If you want to reproduce yourself, look at the demo hw and look at foreman-nuc2.example.com host. The power switcher gives up after 3 seconds, while in legacy UI it gives up much much later (like 20 seconds). If you want to learn more about BMC and how to get it setup in your dev setup, see Tutorial: howto setup BMC (IPMI) for development
Thanks @Marek_Hulan, adding this to my backlog,
if the BMC can be performed via ajax call after the new host page loads,
then we could wait as long as we need to get the response, and indicate a spinner instead of a false status, how does it sound?
maybe we could look on the BMC status and if it’s still empty, keep requesting until we get new data for it, with reasonable timeout.