Timed out reading data from server

Hello All,

I am trying to create VM using hammer and getting below error:

[ERROR 2020-08-13T16:33:19 API] Timed out reading data from server
[ERROR 2020-08-13T16:33:19 Exception] Error: Timed out reading data from server
[ERROR 2020-08-13T16:33:19 Exception]

RestClient::Exceptions::ReadTimeout (Timed out reading data from server):
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:733:in `rescue in transmit'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:642:in `transmit'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:67:in `post'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.3.0/lib/apipie_bindings/api.rb:327:in `call_client'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.3.0/lib/apipie_bindings/api.rb:240:in `http_call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.3.0/lib/apipie_bindings/api.rb:190:in `call_action'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.3.0/lib/apipie_bindings/api.rb:185:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.3.0/lib/apipie_bindings/resource.rb:21:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2/lib/hammer_cli/apipie/command.rb:53:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman-0.19.6/lib/hammer_cli_foreman/commands.rb:188:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2/lib/hammer_cli/apipie/command.rb:34:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2/lib/hammer_cli/abstract.rb:76:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2/lib/hammer_cli/abstract.rb:76:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2/lib/hammer_cli/abstract.rb:76:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2/bin/hammer:147:in `<top (required)>'
    /usr/bin/hammer:23:in `load'
    /usr/bin/hammer:23:in `<main>'

If anybody can help?
Thanks,
Aditya

Hey, it pretty much captures what happened. The request timed out when reading a response. Investigate your network, firewall etc.

I’m having the same problem and have been having it for over a year. Thing is the host still gets created. Does you host get created? I’ve got a ticket open with Red Hat but so far they haven’t been able to figure it out.

For hammer timeout, I’d start with setting it to longer value or -1 (forever): https://access.redhat.com/solutions/1274723

  • Edit /etc/hammer/cli.modules.d/foreman.yml :
  1 :foreman:
  2   # Enable/disable foreman commands
  3   :enable_module: true
  4 
  5   # Your foreman server address
  6   :host: 'https://localhost/'
  7 
  8   # Check API documentation cache status on each request
  9   #:refresh_cache: false
 10 
 11   # API request timeout. Set to -1 for no timeout
 12   :request_timeout: -1 #seconds  

I set it to -1 and I’m still getting a timeout error. So far Red Hat support has not been much help. They actually asked me if I really needed to use hammer cli. They said that the machine isn’t provisioning but the machine gets built and the finish template runs to completion.

content of /etc/hammer/cli.modules.d/foreman.yml

:foreman:
  # Enable/disable foreman commands
  :enable_module: true

  # Your foreman server address
  :host: 'https://serverName'

  # API request timeout. Set to -1 for no timeout
  :request_timeout: -1 #seconds

:ssl:
  :ssl_ca_file: '/etc/pki/katello/certs/katello-server-ca.crt'

Output from hammer command

[ERROR 2020-08-24T08:42:07 API] Timed out reading data from server
[DEBUG 2020-08-24T08:42:07 API] ""
[DEBUG 2020-08-24T08:42:07 Exception] Using exception handler HammerCLIForeman::ExceptionHandler#handle_general_exception
[ERROR 2020-08-24T08:42:07 Exception] Error: Timed out reading data from server
Could not create the host:
  Error: Timed out reading data from server
[ERROR 2020-08-24T08:42:07 Exception]

RestClient::Exceptions::ReadTimeout (Timed out reading data from server):
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:733:in `rescue in transmit'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:642:in `transmit'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:145:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/request.rb:52:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.2/lib/restclient/resource.rb:67:in `post'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.3.0/lib/apipie_bindings/api.rb:327:in `call_client'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.3.0/lib/apipie_bindings/api.rb:240:in `http_call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.3.0/lib/apipie_bindings/api.rb:190:in `call_action'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.3.0/lib/apipie_bindings/api.rb:185:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-bindings-0.3.0/lib/apipie_bindings/resource.rb:21:in `call'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2.1/lib/hammer_cli/apipie/command.rb:53:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli_foreman-0.19.6.5/lib/hammer_cli_foreman/commands.rb:188:in `send_request'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2.1/lib/hammer_cli/apipie/command.rb:34:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2.1/lib/hammer_cli/abstract.rb:76:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2.1/lib/hammer_cli/abstract.rb:76:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/subcommand/execution.rb:11:in `execute'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:63:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2.1/lib/hammer_cli/abstract.rb:76:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/clamp-1.1.2/lib/clamp/command.rb:132:in `run'
    /opt/theforeman/tfm/root/usr/share/gems/gems/hammer_cli-0.19.2.1/bin/hammer:147:in `<top (required)>'
    /bin/hammer:23:in `load'
    /bin/hammer:23:in `<main>'

Sorry to hear that, let me fix this for you.

It looks like the timeout does work, however it only sets request timeout of the HTTP client library, there are two more timeouts: open and read which are always set to default value (1 minute perhaps). I created a patch that sets all three timeouts to the setting you can set via the yaml via request_timeout. You can apply this change easily on your instance, just locate the api.rb file and patch it.

Try this with setting -1 first, I haven’t tested this myself but I think it should work. If not try some high value like 9999. Get back to me if it works for you.

1 Like

Quick followup, the timeout in hammer is a generic issue and my patch helps to workaround that. Since some operations can be quite slow, particularly SSH timeout when host cannot be reached, we should probably make hammer timeout longer by default.

However culprit of your problem is SSH finish provisioning, this only works if DHCP is under Foreman control:

https://docs.theforeman.org/master/Provisioning_Guide/index-foreman.html#_prerequisites_for_image_based_provisioning

There is a undocumented setting in Administer - Settings - Provision - Use DNS for finish script (out of my head might be little bit different wording) and then if and only if DNS is under Foreman control, Foreman will use hostname instead of provisioning IP address. This is a workaround.

2 Likes

It ended up that I kind of had two problems. First although the system timeout was set to never timeout it was set to 120 seconds in /root/.hammer/cli.modules.d/foreman.yml. This setting supersedes the system setting. The second problem was at the end of the finish template we had a reboot command. Because of this Satellite never knew if the finish template completed successfully or not. To get around this we can either schedule a reboot in the finish template using “poweroff” or “at” or reboot manually after provisioning. Thanks to @lzap and the other guys at Red Hat.

1 Like

Oh the system vs user setting is a nasty one!

I’ve created a snippet which should at least inform whoever edits the template not to use reboot command which is nasty. We did not figure out how to actually nicely close a SSH connection when reboot is issued - this is limitation of our SSH Ruby library we use. poweroff -r +1 does the job nicely. For the record: https://github.com/theforeman/foreman/pull/7949

Thanks for getting back to us!

1 Like