DHCP-entries not being cleaned up properly (Proxy::DHCP::Collision)

Support
1

Problem:
We have a daily ansible-job that creates 8 vm’s using theforeman.foreman.host module. The job first removes the pre-existing vm’s from the day before and then creates 8 new ones.

Everytime we run the job nowadays we are getting the following error-message from ansible when it tries to create a vm.
Create DHCP Settings for xxxxxxxx task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::BadRequest]: 400 Bad Request) for proxy https://xxx:9090/dhcp"

Prior to upgrading from Foreman 3.2.0 and Katello 4.4.0 to Foreman 3.3.0 and Katello 4.5.0 this has never been an issue.

I have tried to manually remove the concerning host-entries from /var/lib/dhcpd/dhcpd.leases and that usually makes us able to complete the job once. The second try I run the job, it fails again like in the problem-description.

Expected outcome:
I expect DHCP-entries to be deleted when a vm is deleted so the same IP-address can be reused again.

Foreman and Proxy versions:
Foreman 3.3.0 and Katello 4.5.0

Distribution and version:
RHEL-8

Other relevant data:
dhcpd-log
dhcpd.log (3.6 KB)

proxy-log:
proxy.log (61.3 KB)

dhcpd.conf

omapi-port 7911;
default-lease-time 43200;
max-lease-time 86400;
not authoritative;
ddns-update-style none;

option domain-name-servers 10.87.1.10,10.87.1.11;
option ntp-servers none;

allow booting;
allow bootp;

option fqdn.no-client-update on;  # set the "O" and "S" flag bits
option fqdn.rcode2 255;
option pxegrub code 150 = text;
option architecture code 93 = unsigned integer 16 ;

set vendor-string = option vendor-class-identifier;
log-facility local7;
include "/etc/dhcp/dhcpd.hosts";

class "httpclients" {
  match if substring(option vendor-class-identifier, 0, 10) = "HTTPClient";
  option vendor-class-identifier "HTTPClient";
  if option architecture = 00:0F {
    filename "http://xxxxx:8000/EFI/grub2/shimia32.efi";
  } else if option architecture = 00:10 {
    filename "http://xxxxx:8000/EFI/grub2/shim.efi";
  }
}

class "pxeclients" {
  match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
  next-server 10.87.1.14;
  if exists user-class and option user-class = "iPXE" {
    filename "http://10.87.1.14:8000/unattended/iPXE";
  } else if option architecture = 00:06 {
    filename "grub2/shimia32.efi";
  } else if option architecture = 00:07 {
    filename "grub2/shim.efi";
  } else {
    filename "pxelinux.0";
  }
}

dhcpd.hosts is empty, so we don’t have any static records there.

Subnet pool-range:

subnet 10.87.1.0 netmask 255.255.255.0 {
  pool
  {
    range 10.87.1.30 10.87.1.250;
  }

  option subnet-mask 255.255.255.0;
  option routers 10.87.1.1;
}