Problem: When running hammer host create, I am seeing the following output:
Could not create the host:
execution expired
Foreman and Proxy versions:
foreman-1.22.0-1.el7.noarch
foreman-proxy-1.22.0-1.el7.noarch
Other relevant data:
Some internet searches have suggested there might be a problem with DNS, but the hostname and IP we are using have been verified to resolve correctly on our Foreman host.
We are seeing the following in /var/log/foreman/production.log:
2019-08-12T12:52:06 [I|app|897b5417] Authorized user admin(Admin User)
2019-08-12T12:52:06 [I|app|897b5417] Current user set to admin (admin)
2019-08-12T12:52:10 [W|app|897b5417] Action failed
Net::Error: execution expired
/usr/share/foreman/lib/net/dns.rb:62:in `rescue in lookup'
/usr/share/foreman/lib/net/dns.rb:19:in `lookup'
/usr/share/foreman/lib/net/dns.rb:87:in `dns_lookup'
/usr/share/foreman/lib/net/dns/forward_record.rb:22:in `conflicts'
/usr/share/foreman/lib/net.rb:23:in `conflicting?'
/usr/share/foreman/app/models/concerns/orchestration/dns.rb:99:in `block in queue_remove_dns_co
nflicts'
/usr/share/foreman/app/models/concerns/orchestration/dns.rb:98:in `each'
/usr/share/foreman/app/models/concerns/orchestration/dns.rb:98:in `queue_remove_dns_conflicts'
/usr/share/foreman/app/models/concerns/orchestration/dns.rb:59:in `queue_dns'
The actual work for DNS is done by the Smart Proxy with the DNS feature for the assigned Subnet. Can you look into and provide the corresponding Smart Proxy logs (/var/log/foreman-proxy/proxy.log)? This should provide more details about the failure.
The problem is that Foreman core application performs DNS query to check if the new/changed DNS entry already exists. It does that via OS-level DNS resolver (/etc/resolv.conf) and if you misconfigured that you can run into timeouts.
Foreman also performs DNS lookups aganst Subnet DNS smart proxy in some cases, but in this case (and the stacktrace confirms) it’s the OS resolver which timeouts.
We are improving this code for 1.24 so it’s more configurable with multiple DNS query attempts, today it’s just a single DNS query and a single timeout (which is configurable tho).
2019-08-13T07:54:44 3d71d88f [I] Started GET /serverName
2019-08-13T07:54:44 3d71d88f [I] Finished GET /serverName with 200 (0.31 ms)
2019-08-13T07:54:44 3d71d88f [I] Started GET /172.x.0.0/mac/
2019-08-13T07:54:44 3d71d88f [E] No DHCP record for MAC 172.x.0.0/ found
2019-08-13T07:54:44 3d71d88f [I] Finished GET /172.x.0.0/mac/ with 404 (0.45 ms)
2019-08-13T07:54:44 3d71d88f [I] Started GET /172.x.0.0/ip/172.x.x.x
2019-08-13T07:54:44 3d71d88f [E] No DHCP records for IP 172.x.0.0/172.x.x.x found
2019-08-13T07:54:44 3d71d88f [I] Finished GET /172.x.0.0/ip/172.x.x.x with 404 (0.41 ms)
2019-08-13T07:54:44 3d71d88f [I] Started GET /172.x.0.0/mac/
2019-08-13T07:54:44 3d71d88f [E] No DHCP record for MAC 172.x.0.0/ found
2019-08-13T07:54:44 3d71d88f [I] Finished GET /172.x.0.0/mac/ with 404 (0.45 ms)
2019-08-13T07:54:44 3d71d88f [I] Started GET /172.x.0.0/ip/172.x.x.x
2019-08-13T07:54:44 3d71d88f [E] No DHCP records for IP 172.x.0.0/172.x.x.x found
2019-08-13T07:54:44 3d71d88f [I] Finished GET /172.x.0.0/ip/172.x.x.x with 404 (0.4 ms)
If I’m on the foreman server, I can nslookup both forward and reverse without issues. They return almost immediately (.015s).
Any other ideas or how to get more debugging to help figure out what’s going on?
I know this is confusing, let me explain and put this into our docs.
During DNS record creation, Foreman performs conflict DNS lookups to verify that hostname is not in active use. This check will be performed against one of the following DNS servers:
system-wide resolver if Adminster - Setting - query_local_nameservers is set or
nameserver(s) defined in the Subnet associated with the Host or
authoritative NS queried from SOA from the Domain name associated with the Host
When experiencing timeouts during DNS conflict resolutions, check the following:
Subnet nameservers must be reachable from the Foreman server
Domain name must have a SOA record available from the Foreman server
System resolver (/etc/resolv.conf) must have valid and working configuration