I'm running a Smart Proxy, which is managing DHCP/TFTP for 125 subnets.
I'm trying to figure out what would cause the DHCP record creation to fail,
when switching a host from one subnet to another. When you switch a NIC's
subnet to one with a DHCP smart proxy, the DHCP record gets automatically
created. I was using this to migrate hosts from a generic 10/8 network
(with no DHCP or TFTP proxies configured) to more specific 10.x.x.x network
with a DHCP+TFTP smart proxy. We previously did not use Foreman for
provisioning, and no DHCP/TFTP proxies were defined for the generic 10/8
network.
This was working fine for roughly the first thousand hosts. I'd move the
host to the new subnet, and the corresponding DHCP record was created on
the DHCP server. However, suddenly I'm starting to see errors when I try
to switch hosts to the subnets with the smart proxy. This happens both
from the API, and from the Web UI.
Using the API (from a custom script), I see errors like this (I changed the
IP's and hostnames):
{u'error': {u'message': u'ERF12-4395 [ProxyAPI::ProxyException]: Unable to
retrieve DHCP entry for 10.77.136.211 ([RestClient::RequestTimeout]:
Request Timeout) for proxy https://foreman.example.org:8443/dhcp'}}
and
{u'error': {u'errors': {u'base': [u'Create DHCP Settings for
myhost.example.org task failed with the following error: ERF12-6899
[ProxyAPI::ProxyException]: Unable to set DHCP entry
([RestClient::RequestTimeout]: Request Timeout) for proxy
https://foreman.example.org:8443/dhcp']}, u'id': 35660, u'full_messages': [u'Create
DHCP Settings for myhost.example.org task failed with the following error:
ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry
([RestClient::RequestTimeout]: Request Timeout) for proxy
https://foreman.example.org:8443/dhcp']}}
The first looks like an error retrieving information on the record, and the
second looks like an error updating the record.
From the Web UI, I'll see something like this:
Create DHCP Settings for myhostname.example.org task failed with the
following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP
entry ([RestClient::RequestTimeout]: Request Timeout) for proxy https:
//smartproxy.example.org:8443/dhcp
More often from the Web UI though, what happens is I go in to edit the
host's primary interface, change the subnet, click "OK", then "submit", and
it goes back to the "Host" tab on the "edit" page (ie:
https://foreman.example.org/hosts/myhostname.example.com/edit). No error
message is displayed, but the changed subnet is not saved, and the DHCP
record is not created. In looking at the smart proxy log file, I see lots
of messages like this:
E, [2016-02-11T10:00:05.005071 #15360] ERROR – : Record
10.77.112.0/10.77.126.212 not found
[11/Feb/200:00:05] "GET /10.77.112.0/10.77.126.212 HTTP/1.1" 404 44 62.1812
[11/Feb/2016 10:01:27] "GET /serverName HTTP/1.1" 200 47 0.0006
But I'm not sure if this is a true error, because a successful change looks
like this (with an extra POST at the end):
E, [2016-02-11T10:40:56.269453 #8116] ERROR – : Record
10.12.5.0/10.12.5.102 not found
[11/Feb/2016 10:40:56] "GET /10.12.5.0/10.0.5.102 HTTP/1.1" 404 36 58.2472
[11/Feb/2016 10:41:56] "POST /10.12.5.0 HTTP/1.1" 200 - 59.7205
I'm wondering if it's failing on retrieval, like from the API call, and the
Web UI just doesn't give me an error in the second scenario.
I'm looking for ways to troubleshoot this. Is there anything specific that
could cause it? I wouldn't think I'd have hit any sort of limits, but I'm
not sure. Most confusing is why it would be working fine all week, and
suddenly it's having trouble creating records. The Foreman server itself
is a production server, and is heavily used, but the smart proxy doesn't
have much going on right now.
Some things I've tried:
- Changing the subnet through the API
- Changing the subnet through the WebIU
- Verified DHCP server and foreman-proxy services were running on Smart
Proxy server, and restarted both - Verified that some hosts still work (I've had a change fail, gone back
and tried again, and then it works, but some hosts don't seem to work no
matter how many times I try) - Tried to tie it back to a specific subnet. There are 12 subnets (out of
- where I've seen failures. It doesn't appear to be related to the
subnet though.
I'm running Foreman 1.9.3.
