Saving host always tries to create DHCP lease, leads to conflict error

Hello,

We are currently running into an odd issue with Foreman 1.7.5. When you try
and save a host who's DHCP is managed by a particular smart proxy, it tries
to create a DHCP record and errors due to it already existing:

Create DHCP Settings for <snip> task failed with the following error:
ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry
([RestClient::Conflict]: 409 Conflict) for proxy http://localhost:8000/dhcp

I don't understand why it's trying to create a new record each time, as
there is already a valid lease for that host in /var/lib/dhcpd/dhcp.leases:

host <snip> {
dynamic;
hardware ethernet 52:54:00:c0:53:6e;
fixed-address 192.168.166.74;
supersede server.filename = "pxelinux.0";
supersede server.next-server = c0:a8:a6:5b;
supersede host-name = "<snip>";
}

Here is the log from the smart proxy:

127.0.0.1 - - [02/Jun/2015 17:18:51] "POST /192.168.166.0 HTTP/1.1" 409 50
0.0622
127.0.0.1 - - [02/Jun/2015 17:22:23] "GET /serverName HTTP/1.1" 200 31
0.0013
127.0.0.1 - - [02/Jun/2015 17:22:23] "GET /192.168.166.0/52:54:00:57:d7:b9
HTTP/1.1" 200 184 0.1348
127.0.0.1 - - [02/Jun/2015 17:22:24] "GET /192.168.166.0/192.168.166.74
HTTP/1.1" 200 184 0.1737
E, [2015-06-02T17:22:24.435451 #23152] ERROR – : Record
192.168.166.0/192.168.166.74 already exists
127.0.0.1 - - [02/Jun/2015 17:22:24] "POST /192.168.166.0 HTTP/1.1" 409 50
0.0702

The only other thing I've noticed is that the 'Subnet' field in the Edit
host page is blank for the affected hosts, despite it being set in the host
group and the correct IP being set in the 'IP address' field. After I save
the form and get the conflict error, the Subnet field is then populated
correctly. Setting the 'Subnet' field before saving doesn't seem to make a
difference.

We have two smart proxies which handle different DNS zones, one of which
still seems to be working. The only thing that has changed recently is our
upgrade to Foreman 1.7.5.

Has anyone seen anything like this before?

Thanks,

Andy

> Hello,
>
> We are currently running into an odd issue with Foreman 1.7.5. When you
> try and save a host who's DHCP is managed by a particular smart proxy, it
> tries to create a DHCP record and errors due to it already existing:
>
> Create DHCP Settings for <snip> task failed with the following error:
> ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry
> ([RestClient::Conflict]: 409 Conflict) for proxy
> http://localhost:8000/dhcp
>
>
> a conflict is raised usually when there is a mismatch, between what is
currently deployed vs what you are requesting foreman to do, obviuos cases
is where you have a reservation that conflict (e.g. ip or mac etc) .

further, there is an option to override and force the action (foreman would
delete the old reservation and recreate it) did you try that? I'm missing
the foreman logs here to understand further…

I don't understand why it's trying to create a new record each time, as
> there is already a valid lease for that host in /var/lib/dhcpd/dhcp.leases:
>
> host <snip> {
> dynamic;
> hardware ethernet 52:54:00:c0:53:6e;
> fixed-address 192.168.166.74;
> supersede server.filename = "pxelinux.0";
> supersede server.next-server = c0:a8:a6:5b;
> supersede host-name = "<snip>";
> }
>
> Here is the log from the smart proxy:
>
> 127.0.0.1 - - [02/Jun/2015 17:18:51] "POST /192.168.166.0 HTTP/1.1" 409
> 50 0.0622
> 127.0.0.1 - - [02/Jun/2015 17:22:23] "GET /serverName HTTP/1.1" 200 31
> 0.0013
> 127.0.0.1 - - [02/Jun/2015 17:22:23] "GET /192.168.166.0/52:54:00:57:d7:b9
> HTTP/1.1" 200 184 0.1348
> 127.0.0.1 - - [02/Jun/2015 17:22:24] "GET /192.168.166.0/192.168.166.74
> HTTP/1.1" 200 184 0.1737
> E, [2015-06-02T17:22:24.435451 #23152] ERROR – : Record
> 192.168.166.0/192.168.166.74 already exists
> 127.0.0.1 - - [02/Jun/2015 17:22:24] "POST /192.168.166.0 HTTP/1.1" 409
> 50 0.0702
>
> The only other thing I've noticed is that the 'Subnet' field in the Edit
> host page is blank for the affected hosts, despite it being set in the host
> group and the correct IP being set in the 'IP address' field. After I save
> the form and get the conflict error, the Subnet field is then populated
> correctly. Setting the 'Subnet' field before saving doesn't seem to make a
> difference.
>
thats odd, are you using org/locations? make sure the subnet belongs to the
host org/location (and domain).

··· On Tue, Jun 2, 2015 at 7:29 PM, Andy Taylor wrote:

We have two smart proxies which handle different DNS zones, one of which
still seems to be working. The only thing that has changed recently is our
upgrade to Foreman 1.7.5.

Has anyone seen anything like this before?

Thanks,

Andy


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Hello,

> We are currently running into an odd issue with Foreman 1.7.5. When you try
> and save a host who's DHCP is managed by a particular smart proxy, it tries
> to create a DHCP record and errors due to it already existing:
>

While it is possible to define the same DHCP range in Foreman, it's
usually good practice to select a range from outside the pool defined in
the installer, but still in the subnet. For the example above, it is
recommended to define the DHCP range from 10.0.0.1 to 10.0.0.99 in the
Foreman UI which gives the following IP address distribution:

  • 10.0.0.1 - 10.0.0.99 - addresses reserved during bare-metal provisioning
    by Foreman
  • 10.0.0.100 - 10.0.200 - addresses for dynamic clients in the subnet
    (discovered hosts, unmanaged hosts)

http://theforeman.org/manuals/1.8/#3.2.3InstallationScenarios

··· -- Later, Lukas #lzap Zapletal

Thank you for the suggestions. I've since found that not all hosts managed
by that DHCP smart proxy exhibit the same behaviour - some fail to save due
to a DHCP conflict, some save fine. I've diffed the hosts table entries for
a host that works and a host that doesn't. The main differences seem to be
that for a host that fails, the following rows are NULL in the hosts table:

subnet_id: NULL
ptable_id: NULL
medium_id: NULL

Which seems odd to me, as both of these hosts would have been provisioned
by PXE/kickstart, so those values would need to have been set when the host
was provisioned. I've checked a DB backup prior to the upgrade to 1.7.5 and
it seems to be the same there, so I don't think it's the upgrade.

I've also double checked the dhcpd.leases file for a conflicting IP
address/MAC address and there don't appear to be any.

@Ohad: Both the subnet and the domain belong to the correct
organization/location. I can't see the override option you mentioned in the
interface. Here is a pastebin of the production.log output when I try to
save a broken host:

http://pastebin.com/7HTPAzSs

@Lukas: We already specify a limited start/end inside the affected subnet,
specifically to avoid conflicting with statically configured hosts, so I
don't think it is a conflict with another host on the subnet.

Thanks,

Andy

··· On Wednesday, 3 June 2015 08:50:39 UTC+1, Lukas Zapletal wrote: > > Hello, > > > We are currently running into an odd issue with Foreman 1.7.5. When you > try > > and save a host who's DHCP is managed by a particular smart proxy, it > tries > > to create a DHCP record and errors due to it already existing: > > > > While it is possible to define the same DHCP range in Foreman, it's > usually good practice to select a range from outside the pool defined in > the installer, but still in the subnet. For the example above, it is > recommended to define the DHCP range from 10.0.0.1 to 10.0.0.99 in the > Foreman UI which gives the following IP address distribution: > > * 10.0.0.1 - 10.0.0.99 - addresses reserved during bare-metal provisioning > by Foreman > * 10.0.0.100 - 10.0.200 - addresses for dynamic clients in the subnet > (discovered hosts, unmanaged hosts) > > http://theforeman.org/manuals/1.8/#3.2.3InstallationScenarios > > -- > Later, > Lukas #lzap Zapletal >

Just a quick update, I think I've managed to fix this. Having had a dig
through the code I thought it was probably the NULL value for subnet_id
which was causing the issue, and indeed any host with that setting set to
NULL had the same problem. I've got no idea how they came to be in that
state, but they are all quite old so I'm assuming some sort of
misconfiguration on my part a few months ago.

Anyway, I couldn't set the subnet via the interface as it would keep
getting stuck on the DHCP lease, so I had to change it in the database for
the affected hosts:

UPDATE hosts SET subnet_id = '<subnet id>' WHERE name = <hostname>';

The host edit form can then be saved as per normal.

Thanks for the help,

Andy

··· On Wednesday, 3 June 2015 09:57:13 UTC+1, Andy Taylor wrote: > > Thank you for the suggestions. I've since found that not all hosts managed > by that DHCP smart proxy exhibit the same behaviour - some fail to save due > to a DHCP conflict, some save fine. I've diffed the hosts table entries for > a host that works and a host that doesn't. The main differences seem to be > that for a host that fails, the following rows are NULL in the hosts table: > > subnet_id: NULL > ptable_id: NULL > medium_id: NULL > > Which seems odd to me, as both of these hosts would have been provisioned > by PXE/kickstart, so those values would need to have been set when the host > was provisioned. I've checked a DB backup prior to the upgrade to 1.7.5 and > it seems to be the same there, so I don't think it's the upgrade. > > I've also double checked the dhcpd.leases file for a conflicting IP > address/MAC address and there don't appear to be any. > > @Ohad: Both the subnet and the domain belong to the correct > organization/location. I can't see the override option you mentioned in the > interface. Here is a pastebin of the production.log output when I try to > save a broken host: > > http://pastebin.com/7HTPAzSs > > @Lukas: We already specify a limited start/end inside the affected subnet, > specifically to avoid conflicting with statically configured hosts, so I > don't think it is a conflict with another host on the subnet. > > Thanks, > > Andy > > On Wednesday, 3 June 2015 08:50:39 UTC+1, Lukas Zapletal wrote: >> >> Hello, >> >> > We are currently running into an odd issue with Foreman 1.7.5. When you >> try >> > and save a host who's DHCP is managed by a particular smart proxy, it >> tries >> > to create a DHCP record and errors due to it already existing: >> > >> >> While it is possible to define the same DHCP range in Foreman, it's >> usually good practice to select a range from outside the pool defined in >> the installer, but still in the subnet. For the example above, it is >> recommended to define the DHCP range from 10.0.0.1 to 10.0.0.99 in the >> Foreman UI which gives the following IP address distribution: >> >> * 10.0.0.1 - 10.0.0.99 - addresses reserved during bare-metal >> provisioning >> by Foreman >> * 10.0.0.100 - 10.0.200 - addresses for dynamic clients in the subnet >> (discovered hosts, unmanaged hosts) >> >> http://theforeman.org/manuals/1.8/#3.2.3InstallationScenarios >> >> -- >> Later, >> Lukas #lzap Zapletal >> >