No free IPs in second DHCP zone

Problem:
I have two dhcp pools defined. One from the original installer run and a second one using custom_hiera that was added afterwards. The problem is that the later only returns ‘No free IPs in range’ while the original one works just fine with almost identical settings. The generated files are attached below.
The infra pool is the original one. Both subnets are registered with foreman and have a different range for their static allocations so we don’t run into allocation conflicts. The settings for the subnet are also attached below.
Expected outcome:
The DHCP proxy can assign addresses from the second subnet. Same as the first one. I don’t really know what kind of setting I could have missed that stops this from working. Any help would be appreciated.

Foreman and Proxy versions:
1.20.3
Foreman and Proxy plugin versions:
1.20.3
Distribution and version:
Ubuntu 18.04.2
Other relevant data:
dhpcd.conf

# cluster
subnet 172.31.102.0 netmask 255.255.255.0 {
  pool
  {
    range 172.31.102.101 172.31.102.249;
  }

  option subnet-mask 255.255.255.0;
  option routers 172.31.102.250;
}
# infra.domain
subnet 172.31.100.0 netmask 255.255.254.0 {
  pool
  {
    range 172.31.101.101 172.31.101.249;
  }

  option subnet-mask 255.255.254.0;
  option routers 172.31.101.250;
  option domain-search "[infra.domain,domain]";
}

cluster subnet definition in foreman

It looks correct. Where do you get that error? In Foreman or in the DHCP log itself? Can look at the Foreman, Foreman Proxy and DHCP logs to see what’s going on?

The error comes from the foreman-proxy log. The logs in general are not all that helpful sadly but I attached them below. The foreman-local isc-dhcp-server does not spit out any logs at all as far as I can tell. But the dhcpd works just fine for the second subnet. It’s just that foreman cannot claim any addresses. Which causes a bit of an issue with DNS for that block.

I also just edited the /etc/foreman-proxy/settings.d/dhcp.yml to query the second subnet, just in case. But that didn’t cause any changes.

foreman/production.log

2019-11-05T11:46:50 [I|app|] Started POST "/subnets/freeip" for 10.110.5.34 at 2019-11-05 11:46:50 +0000
2019-11-05T11:46:50 [I|app|7512e] Processing by SubnetsController#freeip as JSON
2019-11-05T11:46:50 [I|app|7512e]   Parameters: {"subnet_id"=>"6", "host_mac"=>"", "taken_ips"=>["", ""]}

foreman-proxy/proxy.log

2019-11-05T11:46:50 3d4eb815 [I] [2019-11-05 11:46:50.307 #6447]     1 -- Started GET /172.31.102.0/unused_ip from=172.31.102.10&to=172.31.102.100
2019-11-05T11:46:50 3d4eb815 [D] [2019-11-05 11:46:50.308 #6447]     0 -- verifying remote client 172.31.100.1 against trusted_hosts ["foreman.infra.domain"]: verifying remote client 172.31.100.1 against trusted_hosts ["foreman.infra.domain"]
2019-11-05T11:46:50 3d4eb815 [W] [2019-11-05 11:46:50.309 #6447]     2 -- No free IPs in range 172.31.102.10..172.31.102.100.: No free IPs in range 172.31.102.10..172.31.102.100.
2019-11-05T11:46:50 3d4eb815 [I] [2019-11-05 11:46:50.309 #6447]     1 -- Finished GET /172.31.102.0/unused_ip with 200 (1.53 ms)

I also tried deleting the subnet and re-importing it from the smart proxy via discovery. That didn’t help either. I thought it might have been an issue as the subnet was created via hammer. But it did register correctly with the DHCP proxy in the first place so the chances were slim.

Interesting. Can you add some debug statements into:

Interested in:

start_address_i, num_of_addresses and then possible_ip each loop possible even record_ips (which can be a lengthy array)