DNS Smart proxy times out

James_Shewey · January 29, 2018, 5:22pm

Hi all,

I’m having an issue with the infoblox DNS proxy. When I go to create a host, I see it trying to create a record, but I get back the error:

Create IPv4 DNS record for doug-alred.example.com task failed with the following error: ERF12-2357 [ProxyAPI::ProxyException]: Unable to set DNS entry ([RestClient::Exceptions::ReadTimeout]: Timed out reading data from server) for proxy https://foreman.example.com:9090/dns

However, I do see that the record gets created inside of the Infoblox. If I set the Smart Proxy to HTTP mode and run a tcpdump, I can see the request and response headers:

Request headers:

POST /dns/ HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
User-Agent: rest-client/2.0.1 (linux-gnu x86_64) ruby/2.2.2p95
X-Request-Id: ca7db6e7fe4d5a1a729d9ee28e1a8cbae66cdf7571e3e03b7ec758bd98a9c9b8
Content-Length: 59
Content-Type: application/x-www-form-urlencoded
Host: foreman.example.com:8000

fqdn=juana-porraz.example.com&value=192.168.1.189&type=A

Response headers:

HTTP/1.1 200 OK 
Content-Type: text/html;charset=utf-8
Content-Length: 0
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
X-Frame-Options: SAMEORIGIN
Server: 
Date: Wed, 24 Jan 2018 17:24:48 GMT
Connection: Keep-Alive

I made sure to update to the 0.0.6 Gem if the Infoblox DNS proxy, but that did not seem to help. Any other ideas what I can look at here?

Thanks!
James

James_Shewey · January 30, 2018, 9:50pm

Well, this appears to be a bug in foreman where responses without a body attempt to parse a non-existent body and error out. While that is RFC compliant for JSON, there should probably be some sort of content returned in the body by the smart-proxy, but I’m not sure how to make it do that…

trinaryouroboros · February 1, 2018, 12:40pm

Just reporting, I’m getting the same issue:

So, there’s three problems:

While it does create an A record, it should be creating a Host record in Infoblox, so that PTR can be automatically configured.
This error occurs due to timeout, because it’s not getting data returned, and returns you back to the host creation screen:

Create IPv4 DNS record for dev-testvm080 task failed with the following error: ERF12-2357 [ProxyAPI::ProxyException]: Unable to set DNS entry ([RestClient::RequestTimeout]: Request Timeout) for proxy http://foreman-proxy:8443/dns

The process by which Foreman is deploying the system is blocked by this DNS failure, ultimately DNS should just be a warning and not interrupt the users deployment, they can always manually add DNS if necessary:

completed - Set up compute instance dev-testvm080
completed - Query instance details for dev-testvm080
completed - Create DHCP Settings for dev-testvm080
running - Create IPv4 DNS record for dev-testvm080
pending - Deploy TFTP PXEGrub2 config for dev-testvm080
pending - Deploy TFTP PXELinux config for dev-testvm080
pending - Deploy TFTP PXEGrub config for dev-testvm080
pending - Fetch TFTP boot files for dev-testvm080

James_Shewey · February 1, 2018, 5:46pm

Great points. Created issues #:

22486
22487
22488

James_Shewey · February 5, 2018, 7:39pm

After some investigation, I found an upstream bug and have come up with an (ugly) fix for this. I submitted a Pull Request for it, but in the interim,

Edit /usr/share/foreman/lib/proxy_api/resource.rb and change line 10 to:

@connect_params = {:read_timeout => nil, :open_timeout => 10,

This resolves the issue for me, and I believe it should resolve it for you as well.

trinaryouroboros · February 6, 2018, 1:12pm

I’m using 1.15, that line originally reads:

@connect_params = {:timeout => Setting[:proxy_request_timeout], :open_timeout => 10, :headers => { :accept => :json },
                  :user => args[:user], :password => args[:password]}

If I change to read_timeout => nil, I get the same error, if I change to timeout => nil, it forever hangs on the dns creation part

James_Shewey · February 6, 2018, 5:16pm

According to the docs, there should be no functional difference between those two. If read_timeout is not set, it will be set to the timeout value and if neither is set this will default to 60 seconds.

After the change, did you restart the httpd and foreman services? I found I needed to restart these to apply the change.

The version shouldn’t matter, but for what it is worth, I am on 1.16. It still took a very long time for it to work, so give it a good while, so if you bailed out without waiting a full 60 seconds after setting timeout to 0, try that again and make sure to be patient. It will still take a very long time to creation to finish.

trinaryouroboros · February 7, 2018, 12:57pm

With read_timeout, the cursor spins as if busy, and hangs on the DNS creation part, then after 60 seconds I get:

Create IPv4 DNS record for dev-testvm087.mydomain.net task failed with the following error: ERF12-2357 [ProxyAPI::ProxyException]: Unable to set DNS entry ([RestClient::RequestTimeout]: Request Timeout) for proxy http://foreman-proxy.mydomain.net:8443/dns

With timeout value, instead, it gets to DNS creation part, and cursor is not busy, it sits there forever…no updates on the screen, nothing. I’ve been waiting 15 minutes now.

In both cases, the DNS A record gets created if dhcp_infoblox.yml option is set to fixedaddress, or it will create a host record if option set to host.

I’m using passenger, if it makes a difference…

James_Shewey · February 7, 2018, 4:34pm

I am also using passenger. I did also upgrade the infoblox gem to 0.0.6 - not sure if that makes a difference.

lzap · February 7, 2018, 4:41pm

I just replied in the PR, I suggest to have options for all the three timeout settings. When set to 0 then use nil so users can essentially turn this off.

trinaryouroboros · February 7, 2018, 5:11pm

Looks like that shouldn’t be a problem:

$ gem list|grep infoblox
infoblox (2.0.4)
smart_proxy_dhcp_infoblox (0.0.7)
smart_proxy_dns_infoblox (0.0.6)

I’m not sure why I’m getting different behavior when setting the timeout.

lzap · June 12, 2019, 8:49am

Hello, I somehow managed to miss this thread, then my review of your patch was a bit offtopic. Sorry about that.

Two of our customers ran into this issue, I am unable to reproduce however after carefully reviewing Infoblox DNS module codebase, I believe that return value from the module which comes from Faraday gem used in Infoblox gem can cause Webrick to return Content-Length either incorrectly or something else.

I created a patch which fixes the return value according to the contract, I hope this helps. Get back to me if you can confirm:

github.com/theforeman/smart_proxy_dns_infoblox

Add explicit return statements

theforeman:master ← lzap:missing-nils

opened 07:54AM - 12 Jun 19 UTC

lzap

+2 -0

Foreman proxy DNS modules have a contract which require them to return return ni…l value when request was processed successfully. The Infoblox DNS module does not do that and some Infoblox instances can cause the code to return some object (e.g. stream) which is causing the Rack stack to keep the connection open causing HTTP stale connection causing read timeout in Foreman application. I can't verify this fixes the issue, however check other DNS modules in Smart Proxy - they do return `nil` explicitly. Search for method `do_create` in the core codebase. A nasty one, if this is confirmed by the customer I will create a warning in the log if a module does not return `nil` value. Or we can discuss changing this poor contract.

I closed some of your reports, feel free to reopen them if you feel it’s still valid. Thanks!