Deleting a provisionned host, then rediscover and reprovision fails at DHCP

Problem:
When I delete a host and rediscover it then reprovision with the same DNS name, dhcp proxy fails by saying duplicate entry.

I have to edit /var/lib/dhcpd/dhcpd.leases and manually remove the entry for the mac.

Expected outcome:

When rebuilding or recreating, dhcp is able to either reuse a still valid dhcp lease or at least overwrite the previous information without breaking.

Foreman and Proxy versions:

Foreman 2.4 with Katello 4.0

Distribution and version:

Installed on RHEL 7

You should configure your DHCP not to intersect with DHCP reservation range in foreman.

This is documented, granted this is buried down at the two places:

https://theforeman.org/manuals/2.4/index.html#3.2.3InstallationScenarios

and

https://theforeman.org/plugins/foreman_discovery/15.0/index.html

I am taking this into the new documentation:

From the new paragraph proposal:

Ensure that DHCP range of all subnets where discovery is planned do not intersect with DHCP lease pool configured for the managed DHCP service.
DHCP range is set in web interface while lease pool range is set via {foreman-installer}.
For example, in a 10.1.0.0/16 network range 10.1.0.0 to 10.1.127.255 could be allocated for leases and 10.1.128.0 to 10.1.255.254 could be allocate for reservations.

Ok so the Discovery range must be distinct from the provisioning range?

I discover hosts using PXE Boot and the Foreman DHCP is the one responsible for DHCP on discovery and then Provisioning on the same subnet with same Foreman DHCP.

From what I understand, the range given when installing Foreman with ‘–foreman-proxy-dhcp-range’ would be used for provisioning and the range given in the web interface would be used for Discovery?

So for Provisioning using Foreman I should create a Subnet with IPAM set to DHCP and the range configured so as to not overlap the range I put in ‘–foreman-proxy-dhcp-range’, is that right?

Thank you.

Yes you got it right, that is the range set by our installer, its called “dhcp lease pool range”, the other is reservation range where Foreman creates host dynamic reservations. No overlap means no conflicts.

I changed the subnet IPAM range so there is no overlap with ‘–foreman-proxy-dhcp-range’.

  • Deleted the provisioned host.
  • Reboot PXE, host is discovered with IP in range set by ‘–foreman-proxy-dhcp-range’
  • Try to provision host, fails with:
    Create DHCP Settings for mac00900b5d3416.baremetal.xxx task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://foreman.xxx.lan:9090/dhcp

It did try to use an IP address from the newly modified range not overlaping with the discovery range.

In leases, an entry for the hostname is already there from the discovery:

host mac00900b5d3416.baremetal.xxx {
dynamic;
hardware ethernet 00:90:0b:5d:34:16;
fixed-address 10.45.222.101;
supersede server.filename = “pxelinux.0”;
supersede server.next-server = c0:a8:03:85;
supersede host-name = “mac00900b5d3416.baremetal.xxx”;
}

It should be assigning 10.45.222.52 as a new provisioning IP to the same hostname and same mac

If I stop dhcpd, remove the lease and restart dhcpd, then I can provision

Please paste the whole leases file, at least the block when the node gets a lease, then it gets provisioned.

host mac00900b5d3416.baremetal.xxx

This is not a epheremal lease, this is a (dynamic) reservation. It is a leftover probably.

Also one important bit: the subnet must have DHCP IPAM, if you have DB or Manual then of course discovery will not change the IP and will attempt with the same IP unless you change it.

With the node I am working with now (blade 5), I deleted it from ‘All Hosts’.

What is left in dhcpd.leases is:

host blade6.baremetal.xxx {
dynamic;
hardware ethernet 00:yy:yy:5d:34:16;
fixed-address 10.45.222.176;
supersede server.filename = “pxelinux.0”;
supersede server.next-server = 0a:2d:de:01;
supersede host-name = “blade6.baremetal.xxx”;
}
server-duid “\000\001\000\001(%\201\357RT\000\000s\266”;

host blade5.baremetal.xxx {
dynamic;
hardware ethernet 00:yy:yy:88:ce:11;
fixed-address 10.45.222.103;
supersede server.filename = “pxelinux.0”;
supersede server.next-server = 0a:2d:de:01;
supersede host-name = “blade5.baremetal.xxx”;
}

When I rediscover this host, I succeeds but gets an IP from another DHCP server. dhcpd.leases stays the same.

When I provision it by setting the same hostname on the same interface (same MAC) I get error:

Create DHCP Settings for blade5.baremetal.xxx task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://foreman.xxx.lan:9090/dhcp

Content of dhcp.leases is still the same, subnet I am using to provision is:

You are seeing the error because Foreman attempted to create a dynamic host reservation with host name blade5.baremetal.xxx which is already present. Delete it or choose a different name.

That’s what I began the initial post with.

I delete a host from the Foreman ‘Hosts’ page and it leaves that entry in the DHCPD leases.

When I delete a host I want all traces of it to be deleted.

I will try again, but I think that I get this error even if I provision with another hostname because of that entry in DHCPD leases.

Oh sorry I missed that one. Well, this works out of box, can you test it again and tail the leases file and paste it here?

Keep in mind that ISC DHCP keeps the data in “append only” file, it does not physically delete the text from the file, it appends an extra statement with “deleted” keyword which our smart proxy understands and treats this entry as deleted. So you can still see these statements (until dhcpd restart) but there should be no conflict.

You can also use this script:

or this one too:

to perform HTTP GET query on /dhcp/1.2.3.4 to see all hosts and reservations exactly how Foreman sees it. Do this before and after you see the error.

I managed to get query.rb working, had to modify paths to the .pem but get connection refused:

1.45.222.1 is the IP where DHCP is served by Foreman, I also tried with 10.45.222.0 but same result.

[root@foreman lanner]# find / | grep ssl/certs
find: ‘/proc/17707’: No such file or directory
/etc/ssl/certs
/etc/puppetlabs/puppet/ssl/certs
/etc/puppetlabs/puppet/ssl/certs/foreman.noviflow.lan.pem
/etc/puppetlabs/puppet/ssl/certs/ca.pem
/opt/puppetlabs/puppet/ssl/certs
[root@foreman lanner]# ./query.rb 'https://foreman.noviflow.lan:4567/dhcp/10.45.222.1' -v
./query.rb --verb get --key /etc/puppetlabs/puppet/ssl/private_keys/foreman.noviflow.lan.pem --cert /etc/puppetlabs/puppet/ssl/certs/foreman.noviflow.lan.pem --ca /etc/puppetlabs/puppet/ssl/certs/ca.pem  --verbose https://foreman.noviflow.lan:4567/dhcp/10.45.222.1
Exception: 'Connection refused - connect(2)'

Starting with empty /var/lib/dhcpd/dhcpd.leases:

After PXE DHCP:


[root@foreman lanner]# tail -f -n 100 /var/lib/dhcpd/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.2.5

server-duid "\000\001\000\001(.\356-RT\000\000s\266";

lease 10.45.222.100 {
  starts 3 2021/05/12 19:52:40;
  ends 4 2021/05/13 07:52:40;
  cltt 3 2021/05/12 19:52:40;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
  set vendor-string = "PXEClient:Arch:00000:UNDI:002001";
}

Once discovered:

[root@foreman lanner]# tail -f -n 100 /var/lib/dhcpd/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.2.5

server-duid "\000\001\000\001(.\356-RT\000\000s\266";

lease 10.45.222.100 {
  starts 3 2021/05/12 19:52:40;
  ends 4 2021/05/13 07:52:40;
  cltt 3 2021/05/12 19:52:40;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
  set vendor-string = "PXEClient:Arch:00000:UNDI:002001";
}
lease 10.45.222.100 {
  starts 3 2021/05/12 19:54:42;
  ends 4 2021/05/13 07:54:42;
  cltt 3 2021/05/12 19:54:42;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
}

Launch provisioning:

[root@foreman lanner]# tail -f -n 100 /var/lib/dhcpd/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.2.5

server-duid "\000\001\000\001(.\356-RT\000\000s\266";

lease 10.45.222.100 {
  starts 3 2021/05/12 19:52:40;
  ends 4 2021/05/13 07:52:40;
  cltt 3 2021/05/12 19:52:40;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
  set vendor-string = "PXEClient:Arch:00000:UNDI:002001";
}
lease 10.45.222.100 {
  starts 3 2021/05/12 19:54:42;
  ends 4 2021/05/13 07:54:42;
  cltt 3 2021/05/12 19:54:42;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
}
host mac00900b5d3416.baremetal.lanner {
  dynamic;
  hardware ethernet 00:90:0b:5d:34:16;
  fixed-address 10.45.222.100;
        supersede server.filename = "pxelinux.0";
        supersede server.next-server = c0:a8:03:85;
        supersede host-name = "mac00900b5d3416.baremetal.lanner";
}

Fully provisioned no changes to the file.

image

Delete the host:

No changes to the file.

image

Reset server IPMI, Discovery is a Success, no changes to the file.

Provision host again, no changes to the file:

If I stop DHCP, delete the info on the node after deleting it, then I can get through the whole process with no error like the first run.

You do not see any text appended to the file?

Then do you see an incoming request to the smart proxy? Check proxy.log.

DHCP orchestration (add/remove host records) will only be performed if:

  • host provisioning interface has a subnet associated
  • that subnet has a DHCP Smart Proxy set
  • both subnet and smartproxy belog to the same organization and location as the host

Can you check all above?

Also what can happen is that after Foreman receives facts from puppet or ansible and they report a different network that it was provisioned with, Foreman will udpate hosts subnet. If that subnet does not have DHCP orchestration then nothing will be done. You can turn off this behavior in Administer - Settings.

I only have 1 location, 1 organization.

Subnet has the DHCP proxy and is part of location + organization:

After first discover + provision:

[root@foreman lanner]# curl -s https://foreman.noviflow.lan:9090/dhcp/10.45.222.0 --cert /etc/foreman-proxy/foreman_ssl_cert.pem --key /etc/foreman-proxy/foreman_ssl_key.pem
{"reservations":[{"name":"mac00900b5d3416.baremetal.lanner","ip":"10.45.222.100","mac":"00:90:0b:5d:34:16","subnet":"10.45.222.0/255.255.255.0","type":"reservation","deleteable":true,"hostname":"mac00900b5d3416.baremetal.lanner","hardware_type":"ethernet","filename":"pxelinux.0","nextServer":"10.45.222.1"}],"leases":[{"name":"lease-00900b5d3416","ip":"10.45.222.100","mac":"00:90:0b:5d:34:16","subnet":"10.45.222.0/255.255.255.0","starts":"2021-05-11 22:24:42 UTC","ends":"2021-05-12 10:24:42 UTC","state":"active","type":"lease","cltt":"2021-05-11 22:24:42 UTC","next_state":"free","hardware_type":"ethernet"}]}
[root@foreman lanner]# cat /var/lib/dhcpd/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.2.5

server-duid "\000\001\000\001(%\201\357RT\000\000s\266";

lease 10.45.222.100 {
  starts 2 2021/05/11 22:22:27;
  ends 3 2021/05/12 10:22:27;
  cltt 2 2021/05/11 22:22:27;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
  set vendor-string = "PXEClient:Arch:00000:UNDI:002001";
}
lease 10.45.222.100 {
  starts 2 2021/05/11 22:24:42;
  ends 3 2021/05/12 10:24:42;
  cltt 2 2021/05/11 22:24:42;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
}
host mac00900b5d3416.baremetal.lanner {
  dynamic;
  hardware ethernet 00:90:0b:5d:34:16;
  fixed-address 10.45.222.100;
        supersede server.filename = "pxelinux.0";
        supersede server.next-server = 0a:2d:de:01;
        supersede host-name = "mac00900b5d3416.baremetal.lanner";
}

There are errors in ‘proxy.log’ after provisioning, this is provisioning sequence when initated:

2021-05-11T18:26:37 ca2e868b [I] Started GET /dhcp/10.45.222.0
2021-05-11T18:26:37 ca2e868b [I] Finished GET /dhcp/10.45.222.0 with 200 (1.09 ms)
2021-05-11T18:28:36 bfe81e40 [I] Started GET /dhcp/10.45.222.1/mac/00:90:0b:5d:34:16
2021-05-11T18:28:36 bfe81e40 [E] No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found
2021-05-11T18:28:36 bfe81e40 [W] Error details for No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found: <Exception>: No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found
2021-05-11T18:28:36 bfe81e40 [W] No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found: <Exception>: No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found
2021-05-11T18:28:36 bfe81e40 [I] Finished GET /dhcp/10.45.222.1/mac/00:90:0b:5d:34:16 with 404 (1.48 ms)
2021-05-11T18:28:36 bfe81e40 [I] Started GET /dhcp/10.45.222.1/unused_ip mac=00:90:0b:5d:34:16&from=10.45.222.100&to=10.45.222.200
2021-05-11T18:28:36 bfe81e40 [I] Finished GET /dhcp/10.45.222.1/unused_ip with 200 (1.02 ms)
2021-05-11T18:28:36 bfe81e40 [I] Started GET /tftp/serverName
2021-05-11T18:28:36 bfe81e40 [I] Finished GET /tftp/serverName with 200 (0.72 ms)
2021-05-11T18:28:36 bfe81e40 [I] Started GET /dhcp/10.45.222.1/mac/00:90:0b:5d:34:16
2021-05-11T18:28:36 bfe81e40 [E] No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found
2021-05-11T18:28:36 bfe81e40 [W] Error details for No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found: <Exception>: No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found
2021-05-11T18:28:36 bfe81e40 [W] No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found: <Exception>: No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found
2021-05-11T18:28:36 bfe81e40 [I] Finished GET /dhcp/10.45.222.1/mac/00:90:0b:5d:34:16 with 404 (1.28 ms)
2021-05-11T18:28:36 bfe81e40 [I] Started GET /dhcp/10.45.222.1/ip/10.45.222.100
2021-05-11T18:28:36 bfe81e40 [E] No DHCP records for IP 10.45.222.1/10.45.222.100 found
2021-05-11T18:28:36 bfe81e40 [W] Error details for No DHCP records for IP 10.45.222.1/10.45.222.100 found: <Exception>: No DHCP records for IP 10.45.222.1/10.45.222.100 found
2021-05-11T18:28:36 bfe81e40 [W] No DHCP records for IP 10.45.222.1/10.45.222.100 found: <Exception>: No DHCP records for IP 10.45.222.1/10.45.222.100 found
2021-05-11T18:28:36 bfe81e40 [I] Finished GET /dhcp/10.45.222.1/ip/10.45.222.100 with 404 (0.95 ms)
2021-05-11T18:28:37 bfe81e40 [I] Started POST /dhcp/10.45.222.1
2021-05-11T18:28:37 bfe81e40 [I] Next-server option not IPv4, trying to resolve 'foreman.noviflow.lan'
2021-05-11T18:28:37 bfe81e40 [I] Finished POST /dhcp/10.45.222.1 with 200 (270.17 ms)
2021-05-11T18:28:37 bfe81e40 [I] Started POST /dns/
2021-05-11T18:28:37 bfe81e40 [I] Finished POST /dns/ with 200 (624.11 ms)
2021-05-11T18:28:38 bfe81e40 [I] Started POST /dns/
2021-05-11T18:28:38 bfe81e40 [I] Finished POST /dns/ with 200 (143.18 ms)
2021-05-11T18:28:38 bfe81e40 [I] Started POST /tftp/PXELinux/00:90:0b:5d:34:16
2021-05-11T18:28:38 bfe81e40 [I] Finished POST /tftp/PXELinux/00:90:0b:5d:34:16 with 200 (2.06 ms)
2021-05-11T18:28:38 bfe81e40 [I] Started POST /tftp/PXEGrub2/00:90:0b:5d:34:16
2021-05-11T18:28:38 bfe81e40 [I] Finished POST /tftp/PXEGrub2/00:90:0b:5d:34:16 with 200 (2.07 ms)
2021-05-11T18:28:38 bfe81e40 [I] Started POST /tftp/PXEGrub/00:90:0b:5d:34:16
2021-05-11T18:28:38 bfe81e40 [I] Finished POST /tftp/PXEGrub/00:90:0b:5d:34:16 with 200 (1.76 ms)
2021-05-11T18:28:39 bfe81e40 [I] Started POST /tftp/iPXE/00:90:0b:5d:34:16
2021-05-11T18:28:39 bfe81e40 [I] Finished POST /tftp/iPXE/00:90:0b:5d:34:16 with 200 (11.24 ms)
2021-05-11T18:28:39 bfe81e40 [I] Started POST /tftp/fetch_boot_file
2021-05-11T18:28:39 bfe81e40 [I] Finished POST /tftp/fetch_boot_file with 200 (1.19 ms)
2021-05-11T18:28:39 bfe81e40 [I] [22941] Started task ["/usr/bin/wget", "--connect-timeout=10", "--dns-timeout=10", "--read-timeout=60", "--tries=3", "--no-check-certificate", "-nv", "-c", "http://foreman.noviflow.lan/pub/rhel8-install/images/pxeboot/vmlinuz", "-O", "/var/lib/tftpboot/boot/rhel-8-3-lQRyI5Q8tLH6-vmlinuz"]
2021-05-11T18:28:39 bfe81e40 [I] Started POST /tftp/fetch_boot_file
2021-05-11T18:28:39 bfe81e40 [I] Finished POST /tftp/fetch_boot_file with 200 (1.2 ms)
2021-05-11T18:28:39 bfe81e40 [I] [22947] Started task ["/usr/bin/wget", "--connect-timeout=10", "--dns-timeout=10", "--read-timeout=60", "--tries=3", "--no-check-certificate", "-nv", "-c", "http://foreman.noviflow.lan/pub/rhel8-install/images/pxeboot/initrd.img", "-O", "/var/lib/tftpboot/boot/rhel-8-3-lQRyI5Q8tLH6-initrd.img"]
2021-05-11T18:28:39 bfe81e40 [I] Started DELETE /ssh/known_hosts/10.45.222.100
2021-05-11T18:28:39 bfe81e40 [I] Finished DELETE /ssh/known_hosts/10.45.222.100 with 204 (1.64 ms)
2021-05-11T18:28:39 bfe81e40 [I] Started DELETE /ssh/known_hosts/mac00900b5d3416.baremetal.lanner
2021-05-11T18:28:39 bfe81e40 [I] Finished DELETE /ssh/known_hosts/mac00900b5d3416.baremetal.lanner with 204 (0.78 ms)
2021-05-11T18:28:39 bfe81e40 [I] Started DELETE /puppet/ca/autosign/mac00900b5d3416.baremetal.lanner
2021-05-11T18:28:39 bfe81e40 [E] Attempt to remove nonexistent client autosign for mac00900b5d3416.baremetal.lanner
2021-05-11T18:28:39 bfe81e40 [W] Error details for Attempt to remove nonexistent client autosign for mac00900b5d3416.baremetal.lanner: <Exception>: Attempt to remove nonexistent client autosign for mac00900b5d3416.baremetal.lanner
2021-05-11T18:28:39 bfe81e40 [W] Attempt to remove nonexistent client autosign for mac00900b5d3416.baremetal.lanner: <Exception>: Attempt to remove nonexistent client autosign for mac00900b5d3416.baremetal.lanner
2021-05-11T18:28:39 bfe81e40 [I] Finished DELETE /puppet/ca/autosign/mac00900b5d3416.baremetal.lanner with 404 (1.79 ms)
2021-05-11T18:28:39 bfe81e40 [I] Started DELETE /puppet/ca/mac00900b5d3416.baremetal.lanner
2021-05-11T18:28:39 bfe81e40 [I] Finished DELETE /puppet/ca/mac00900b5d3416.baremetal.lanner with 200 (120.89 ms)
2021-05-11T18:28:39 bfe81e40 [I] Started POST /puppet/ca/autosign/mac00900b5d3416.baremetal.lanner
2021-05-11T18:28:39 bfe81e40 [I] Finished POST /puppet/ca/autosign/mac00900b5d3416.baremetal.lanner with 200 (1.01 ms)
2021-05-11T18:28:39 24634fca [I] Started PUT /discovery/10.45.222.100/power/reboot
2021-05-11T18:28:39 24634fca [I] Finished PUT /discovery/10.45.222.100/power/reboot with 200 (75.15 ms)

The address of the interface on the Foreman host is 10.45.222.1 for the discovery/provisioning subnet.
When I created the subnet in Foreman I set it as the ‘Network Address’ because I think that is what goes there:

The format of the error seems weird, is it normal for the lookup to have ‘10.45.222.1’?

After provisioning is over:

curl -s https://foreman.noviflow.lan:9090/dhcp/10.45.222.0 --cert /etc/foreman-proxy/foreman_ssl_cert.pem --key /etc/foreman-proxy/foreman_ssl_key.pem
{"reservations":[{"name":"mac00900b5d3416.baremetal.lanner","ip":"10.45.222.100","mac":"00:90:0b:5d:34:16","subnet":"10.45.222.0/255.255.255.0","type":"reservation","deleteable":true,"hostname":"mac00900b5d3416.baremetal.lanner","hardware_type":"ethernet","filename":"pxelinux.0","nextServer":"10.45.222.1"}],"leases":[{"name":"lease-00900b5d3416","ip":"10.45.222.100","mac":"00:90:0b:5d:34:16","subnet":"10.45.222.0/255.255.255.0","starts":"2021-05-11 22:24:42 UTC","ends":"2021-05-12 10:24:42 UTC","state":"active","type":"lease","cltt":"2021-05-11 22:24:42 UTC","next_state":"free","hardware_type":"ethernet"}]}[root@foreman lanner]# curl at /var/lib/dhcpd/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.2.5

server-duid "\000\001\000\001(%\201\357RT\000\000s\266";

lease 10.45.222.100 {
  starts 2 2021/05/11 22:22:27;
  ends 3 2021/05/12 10:22:27;
  cltt 2 2021/05/11 22:22:27;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
  set vendor-string = "PXEClient:Arch:00000:UNDI:002001";
}
lease 10.45.222.100 {
  starts 2 2021/05/11 22:24:42;
  ends 3 2021/05/12 10:24:42;
  cltt 2 2021/05/11 22:24:42;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
}
host mac00900b5d3416.baremetal.lanner {
  dynamic;
  hardware ethernet 00:90:0b:5d:34:16;
  fixed-address 10.45.222.100;
        supersede server.filename = "pxelinux.0";
        supersede server.next-server = 0a:2d:de:01;
        supersede host-name = "mac00900b5d3416.baremetal.lanner";
}

proxy.log:

2021-05-11T18:42:01 e8ff124b [I] Started GET /tftp/serverName
2021-05-11T18:42:01 e8ff124b [I] Finished GET /tftp/serverName with 200 (0.9 ms)
2021-05-11T18:42:01 e8ff124b [I] Started GET /dhcp/10.45.222.1/mac/00:90:0b:5d:34:16
2021-05-11T18:42:01 e8ff124b [E] No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found
2021-05-11T18:42:01 e8ff124b [W] Error details for No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found: <Exception>: No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found
2021-05-11T18:42:01 e8ff124b [W] No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found: <Exception>: No DHCP record for MAC 10.45.222.1/00:90:0b:5d:34:16 found
2021-05-11T18:42:01 e8ff124b [I] Finished GET /dhcp/10.45.222.1/mac/00:90:0b:5d:34:16 with 404 (1.28 ms)
2021-05-11T18:42:01 e8ff124b [I] Started GET /dhcp/10.45.222.1/ip/10.45.222.100
2021-05-11T18:42:01 e8ff124b [E] No DHCP records for IP 10.45.222.1/10.45.222.100 found
2021-05-11T18:42:01 e8ff124b [W] Error details for No DHCP records for IP 10.45.222.1/10.45.222.100 found: <Exception>: No DHCP records for IP 10.45.222.1/10.45.222.100 found
2021-05-11T18:42:01 e8ff124b [W] No DHCP records for IP 10.45.222.1/10.45.222.100 found: <Exception>: No DHCP records for IP 10.45.222.1/10.45.222.100 found
2021-05-11T18:42:01 e8ff124b [I] Finished GET /dhcp/10.45.222.1/ip/10.45.222.100 with 404 (1.0 ms)
2021-05-11T18:42:01 e8ff124b [I] Started POST /tftp/PXELinux/00:90:0b:5d:34:16
2021-05-11T18:42:01 e8ff124b [I] Finished POST /tftp/PXELinux/00:90:0b:5d:34:16 with 200 (2.3 ms)
2021-05-11T18:42:01 e8ff124b [I] Started POST /tftp/PXEGrub2/00:90:0b:5d:34:16
2021-05-11T18:42:01 e8ff124b [I] Finished POST /tftp/PXEGrub2/00:90:0b:5d:34:16 with 200 (2.82 ms)
2021-05-11T18:42:01 e8ff124b [I] Started POST /tftp/PXEGrub/00:90:0b:5d:34:16
2021-05-11T18:42:01 e8ff124b [I] Finished POST /tftp/PXEGrub/00:90:0b:5d:34:16 with 200 (1.67 ms)
2021-05-11T18:42:01 e8ff124b [I] Started POST /tftp/iPXE/00:90:0b:5d:34:16
2021-05-11T18:42:01 e8ff124b [I] Finished POST /tftp/iPXE/00:90:0b:5d:34:16 with 200 (1.06 ms)
2021-05-11T18:42:01 e8ff124b [I] Started DELETE /puppet/ca/autosign/mac00900b5d3416.baremetal.lanner
2021-05-11T18:42:01 e8ff124b [I] Finished DELETE /puppet/ca/autosign/mac00900b5d3416.baremetal.lanner with 200 (1.12 ms)
2021-05-11T18:47:10 7c911df7 [I] Started GET /dhcp/10.45.222.0
2021-05-11T18:47:10 7c911df7 [I] Finished GET /dhcp/10.45.222.0 with 200 (1.1 ms)

After deleting the host:

[root@foreman lanner]# curl -s https://foreman.noviflow.lan:9090/dhcp/10.45.222.0 --cert /etc/foreman-proxy/foreman_ssl_cert.pem --key /etc/foreman-proxy/foreman_ssl_key.pem
{"reservations":[{"name":"mac00900b5d3416.baremetal.lanner","ip":"10.45.222.100","mac":"00:90:0b:5d:34:16","subnet":"10.45.222.0/255.255.255.0","type":"reservation","deleteable":true,"hostname":"mac00900b5d3416.baremetal.lanner","hardware_type":"ethernet","filename":"pxelinux.0","nextServer":"10.45.222.1"}],"leases":[{"name":"lease-00900b5d3416","ip":"10.45.222.100","mac":"00:90:0b:5d:34:16","subnet":"10.45.222.0/255.255.255.0","starts":"2021-05-11 22:24:42 UTC","ends":"2021-05-12 10:24:42 UTC","state":"active","type":"lease","cltt":"2021-05-11 22:24:42 UTC","next_state":"free","hardware_type":"ethernet"}]}[root@foreman lanner]# curl at /var/lib/dhcpd/dhcpd.leases
# The format of this file is documented in the dhcpd.leases(5) manual page.
# This lease file was written by isc-dhcp-4.2.5

server-duid "\000\001\000\001(%\201\357RT\000\000s\266";

lease 10.45.222.100 {
  starts 2 2021/05/11 22:22:27;
  ends 3 2021/05/12 10:22:27;
  cltt 2 2021/05/11 22:22:27;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
  set vendor-string = "PXEClient:Arch:00000:UNDI:002001";
}
lease 10.45.222.100 {
  starts 2 2021/05/11 22:24:42;
  ends 3 2021/05/12 10:24:42;
  cltt 2 2021/05/11 22:24:42;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 00:90:0b:5d:34:16;
}
host mac00900b5d3416.baremetal.lanner {
  dynamic;
  hardware ethernet 00:90:0b:5d:34:16;
  fixed-address 10.45.222.100;
        supersede server.filename = "pxelinux.0";
        supersede server.next-server = 0a:2d:de:01;
        supersede host-name = "mac00900b5d3416.baremetal.lanner";
}

proxy.log:

2021-05-11T18:49:50 e1307683 [I] Started DELETE /dns/mac00900b5d3416.baremetal.lanner/A
2021-05-11T18:49:51 e1307683 [I] Finished DELETE /dns/mac00900b5d3416.baremetal.lanner/A with 200 (126.38 ms)
2021-05-11T18:49:51 e1307683 [I] Started DELETE /dns/100.222.45.10.in-addr.arpa
2021-05-11T18:49:51 e1307683 [I] Finished DELETE /dns/100.222.45.10.in-addr.arpa with 200 (179.33 ms)
2021-05-11T18:49:51 e1307683 [I] Started GET /tftp/serverName
2021-05-11T18:49:51 e1307683 [I] Finished GET /tftp/serverName with 200 (0.78 ms)
2021-05-11T18:49:51 e1307683 [I] Started DELETE /dhcp/10.45.222.1/mac/00:90:0b:5d:34:16
2021-05-11T18:49:51 e1307683 [I] Finished DELETE /dhcp/10.45.222.1/mac/00:90:0b:5d:34:16 with 200 (0.98 ms)
2021-05-11T18:49:51 e1307683 [I] Started DELETE /tftp/PXELinux/00:90:0b:5d:34:16
2021-05-11T18:49:51 e1307683 [I] Finished DELETE /tftp/PXELinux/00:90:0b:5d:34:16 with 200 (1.18 ms)
2021-05-11T18:49:51 e1307683 [I] Started DELETE /tftp/PXEGrub2/00:90:0b:5d:34:16
2021-05-11T18:49:51 e1307683 [I] Finished DELETE /tftp/PXEGrub2/00:90:0b:5d:34:16 with 200 (1.07 ms)
2021-05-11T18:49:51 e1307683 [I] Started DELETE /tftp/PXEGrub/00:90:0b:5d:34:16
2021-05-11T18:49:51 e1307683 [I] Finished DELETE /tftp/PXEGrub/00:90:0b:5d:34:16 with 200 (1.0 ms)
2021-05-11T18:49:51 e1307683 [I] Started DELETE /tftp/iPXE/00:90:0b:5d:34:16
2021-05-11T18:49:51 e1307683 [I] Finished DELETE /tftp/iPXE/00:90:0b:5d:34:16 with 200 (0.73 ms)
2021-05-11T18:49:51 e1307683 [I] Started DELETE /ssh/known_hosts/10.45.222.100
2021-05-11T18:49:51 e1307683 [I] Finished DELETE /ssh/known_hosts/10.45.222.100 with 204 (1.39 ms)
2021-05-11T18:49:51 e1307683 [I] Started DELETE /ssh/known_hosts/192.168.0.156
2021-05-11T18:49:51 e1307683 [I] Finished DELETE /ssh/known_hosts/192.168.0.156 with 204 (1.0 ms)
2021-05-11T18:49:51 e1307683 [I] Started DELETE /ssh/known_hosts/mac00900b5d3416.baremetal.lanner
2021-05-11T18:49:51 e1307683 [I] Finished DELETE /ssh/known_hosts/mac00900b5d3416.baremetal.lanner with 204 (0.89 ms)
2021-05-11T18:49:52 e1307683 [I] Started DELETE /puppet/ca/autosign/mac00900b5d3416.baremetal.lanner
2021-05-11T18:49:52 e1307683 [E] Attempt to remove nonexistent client autosign for mac00900b5d3416.baremetal.lanner
2021-05-11T18:49:52 e1307683 [W] Error details for Attempt to remove nonexistent client autosign for mac00900b5d3416.baremetal.lanner: <Exception>: Attempt to remove nonexistent client autosign for mac00900b5d3416.baremetal.lanner
2021-05-11T18:49:52 e1307683 [W] Attempt to remove nonexistent client autosign for mac00900b5d3416.baremetal.lanner: <Exception>: Attempt to remove nonexistent client autosign for mac00900b5d3416.baremetal.lanner
2021-05-11T18:49:52 e1307683 [I] Finished DELETE /puppet/ca/autosign/mac00900b5d3416.baremetal.lanner with 404 (1.47 ms)
2021-05-11T18:49:52 e1307683 [I] Started DELETE /puppet/ca/mac00900b5d3416.baremetal.lanner
2021-05-11T18:49:52 e1307683 [I] Finished DELETE /puppet/ca/mac00900b5d3416.baremetal.lanner with 200 (236.89 ms)

Thanks, these lines clearly shows that proxy DHCP module successfully deleted the entry but you say you do not see anything in the leases file. You should see a new statement APPENDED to the file there something like:

host mac00900b5d3416.baremetal.lanner {
  deleted;
}

Couple of questions.

First, can you confirm that when you are deleting the host it has the same name: mac00900b5d3416.baremetal.lanner or do you rename the host after it is provisioned somehow?

Can you increase verbosity of proxy to DEBUG and try again? Paste the deletion log, I am particularly interested in whole request (grep the UUID, in the example above that would be e1307683).

Also, after you delete the host in Foreman and investigate logs, can you also restart the dhcpd server and paste the leases file? The daemon squashes the file on restart, just to be sure.

Finally, can you pastebin whole dhcpd.conf with all the includes? If you are concerned about data, then you can email it to me to lzap a|t redhat.com. DHCP could be configured in a way that an omapi key is allowed to add entries but not to delete them.