DHCP entries never deleted

Problem:
I’m rebuidling my foreman instance and running into problems with the built in DHCP server. I have it setup fine to handle discovery and initial provisioning, but if I try to reprovision the server, I get a DHCP conflict. This also happens if I delete a server and rediscover it, the DHCP entries are never cleaned up. I havent seen any other posts talking about a similar issue, so I’m hoping I just missed a setting and will apprecieate any help.

My work around is to manually delete the DHCP entry through the API

curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem -X "DELETE" https://foreman-1.app.domain:9090/dhcp/10.10.54.0/ip/10.10.54.25

After manually deleting the DHCP entry, I can rebuild the host without issues.

Thanks!

Expected outcome:
I expect foreman to overwrite DHCP entries it manages.

Foreman and Proxy versions:
3.0.0

Foreman and Proxy plugin versions:
foreman-tasks: 5.1.0
foreman_ansible: 6.4.1
foreman_bootdisk: 18.0.0
foreman_discovery: 18.0.0
foreman_puppet: 1.0.3
foreman_remote_execution: 4.7.0
katello: 4.2.0.1.rc3

Distribution and version:
Red Hat Enterprise Linux release 8.4 (Ootpa)

Other relevant data:
Configuration:

foreman-installer \
--foreman-proxy-dhcp=true \
--foreman-proxy-dhcp-gateway=10.10.54.1 \
--foreman-proxy-dhcp-range="10.10.54.10 10.10.54.249" \
--foreman-proxy-dhcp-nameservers="10.10.54.250,10.10.54.251" \
--foreman-proxy-tftp=true \
--enable-foreman-plugin-discovery \
--enable-foreman-proxy-plugin-discovery \
--foreman-proxy-plugin-discovery-install-images=true \
--foreman-proxy-http true \
--foreman-proxy-httpboot=true \
--enable-foreman-plugin-remote-execution \
--enable-foreman-proxy-plugin-remote-execution-ssh \
--enable-foreman-plugin-ansible --enable-foreman-proxy-plugin-ansible \
--enable-foreman-plugin-bootdisk

UI subnet config:

In the UI, I receive this message

Failed to enable centos.app.domain for installation: ["Create DHCP Settings for centos.app.domain task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://foreman-1.app.domain:9090/dhcp", "Failed to perform rollback on Remove DHCP Settings for centos.app.domain - ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://foreman-1.app.domain:9090/dhcp"]

/var/log/foreman-proxy/proxy.log

2021-10-13T22:47:28 43dc6537 [I] Started GET /v2/features 
2021-10-13T22:47:28 43dc6537 [I] Finished GET /v2/features with 200 (226.26 ms)
2021-10-13T22:47:32 f5fa75db [I] Started GET /tftp/serverName 
2021-10-13T22:47:32 f5fa75db [I] Finished GET /tftp/serverName with 200 (0.67 ms)
2021-10-13T22:47:32 f5fa75db [I] Started GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-13T22:47:32 f5fa75db [E] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [W] Error details for No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [W] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [I] Finished GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 404 (1.71 ms)
2021-10-13T22:47:32 f5fa75db [I] Started GET /dhcp/10.10.54.1/ip/10.10.54.233 
2021-10-13T22:47:32 f5fa75db [E] No DHCP records for IP 10.10.54.1/10.10.54.233 found
2021-10-13T22:47:32 f5fa75db [W] Error details for No DHCP records for IP 10.10.54.1/10.10.54.233 found: <Exception>: No DHCP records for IP 10.10.54.1/10.10.54.233 found
2021-10-13T22:47:32 f5fa75db [W] No DHCP records for IP 10.10.54.1/10.10.54.233 found: <Exception>: No DHCP records for IP 10.10.54.1/10.10.54.233 found
2021-10-13T22:47:32 f5fa75db [I] Finished GET /dhcp/10.10.54.1/ip/10.10.54.233 with 404 (0.81 ms)
2021-10-13T22:47:32 f5fa75db [I] Started GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-13T22:47:32 f5fa75db [E] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [W] Error details for No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [W] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [I] Finished GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 404 (0.69 ms)
2021-10-13T22:47:32 f5fa75db [I] Started GET /tftp/serverName 
2021-10-13T22:47:32 f5fa75db [I] Finished GET /tftp/serverName with 200 (0.56 ms)
2021-10-13T22:47:32 f5fa75db [I] Started DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-13T22:47:32 f5fa75db [I] Finished DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 200 (0.53 ms)
2021-10-13T22:47:32 f5fa75db [I] Started POST /dhcp/10.10.54.1 
2021-10-13T22:47:32 f5fa75db [W] Request to create a conflicting DHCP record
2021-10-13T22:47:32 f5fa75db [E] Record 10.10.54.0/10.10.54.233 already exists
2021-10-13T22:47:32 f5fa75db [W] Error details for Record 10.10.54.0/10.10.54.233 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.233 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-13T22:47:32 f5fa75db [W] Record 10.10.54.0/10.10.54.233 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.233 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-13T22:47:32 f5fa75db [I] Finished POST /dhcp/10.10.54.1 with 409 (4.08 ms)
2021-10-13T22:47:32 f5fa75db [I] Started POST /dhcp/10.10.54.1 
2021-10-13T22:47:32 f5fa75db [W] Request to create a conflicting DHCP record
2021-10-13T22:47:32 f5fa75db [E] Record 10.10.54.0/10.10.54.233 already exists
2021-10-13T22:47:32 f5fa75db [W] Error details for Record 10.10.54.0/10.10.54.233 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.233 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-13T22:47:32 f5fa75db [W] Record 10.10.54.0/10.10.54.233 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.233 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-13T22:47:32 f5fa75db [I] Finished POST /dhcp/10.10.54.1 with 409 (1.45 ms)

Set the DHCP pool range as 10-99 and from/to IP range as 100-250. No intersections.

When you discover a host it will get a lease from 10-99, when you provision it it will query our DHCP module to find a free IP in the 100-250 range.

For the record, dhcpd.leases is a append only file, things are deleted there after server performs a squash. Search for the deleted keyword.

Thanks Izap for the configuration change. I just tried it out and I’m still getting the DHCP conflicts

I ran this installer command

foreman-installer  --foreman-proxy-dhcp-range="10.10.54.10 10.10.54.99"

and then updated my app subnet to the 100-249 range

I can use discovery to boot the server, do the initial provision without a problem. If I try to provision an already configured host, I still get the same error in the UI:

Failed to enable centos.app.domain for installation: ["Create DHCP Settings for centos.app.domain task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://foreman-1.app.domain:9090/dhcp", "Failed to perform rollback on Remove DHCP Settings for centos.app.domain - ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://foreman-1.app.domain:9090/dhcp"]

Here’s proxy.log when I run the build command

2021-10-14T23:02:38 b46cdace [I] Started GET /v2/features 
2021-10-14T23:02:38 b46cdace [I] Finished GET /v2/features with 200 (19.0 ms)
2021-10-14T23:02:39 eca0104a [I] Started GET /tftp/serverName 
2021-10-14T23:02:39 eca0104a [I] Finished GET /tftp/serverName with 200 (0.5 ms)
2021-10-14T23:02:39 eca0104a [I] Started GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-14T23:02:39 eca0104a [E] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [W] Error details for No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [W] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [I] Finished GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 404 (1.34 ms)
2021-10-14T23:02:39 eca0104a [I] Started GET /dhcp/10.10.54.1/ip/10.10.54.223 
2021-10-14T23:02:39 eca0104a [E] No DHCP records for IP 10.10.54.1/10.10.54.223 found
2021-10-14T23:02:39 eca0104a [W] Error details for No DHCP records for IP 10.10.54.1/10.10.54.223 found: <Exception>: No DHCP records for IP 10.10.54.1/10.10.54.223 found
2021-10-14T23:02:39 eca0104a [W] No DHCP records for IP 10.10.54.1/10.10.54.223 found: <Exception>: No DHCP records for IP 10.10.54.1/10.10.54.223 found
2021-10-14T23:02:39 eca0104a [I] Finished GET /dhcp/10.10.54.1/ip/10.10.54.223 with 404 (1.15 ms)
2021-10-14T23:02:39 eca0104a [I] Started GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-14T23:02:39 eca0104a [E] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [W] Error details for No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [W] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [I] Finished GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 404 (0.73 ms)
2021-10-14T23:02:39 eca0104a [I] Started GET /tftp/serverName 
2021-10-14T23:02:39 eca0104a [I] Finished GET /tftp/serverName with 200 (0.56 ms)
2021-10-14T23:02:39 eca0104a [I] Started DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-14T23:02:39 eca0104a [I] Finished DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 200 (0.54 ms)
2021-10-14T23:02:39 eca0104a [I] Started POST /dhcp/10.10.54.1 
2021-10-14T23:02:39 eca0104a [W] Request to create a conflicting DHCP record
2021-10-14T23:02:39 eca0104a [E] Record 10.10.54.0/10.10.54.223 already exists
2021-10-14T23:02:39 eca0104a [W] Error details for Record 10.10.54.0/10.10.54.223 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.223 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-14T23:02:39 eca0104a [W] Record 10.10.54.0/10.10.54.223 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.223 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-14T23:02:39 eca0104a [I] Finished POST /dhcp/10.10.54.1 with 409 (4.31 ms)
2021-10-14T23:02:39 eca0104a [I] Started POST /dhcp/10.10.54.1 
2021-10-14T23:02:39 eca0104a [W] Request to create a conflicting DHCP record
2021-10-14T23:02:39 eca0104a [E] Record 10.10.54.0/10.10.54.223 already exists
2021-10-14T23:02:39 eca0104a [W] Error details for Record 10.10.54.0/10.10.54.223 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.223 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-14T23:02:39 eca0104a [W] Record 10.10.54.0/10.10.54.223 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.223 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-14T23:02:39 eca0104a [I] Finished POST /dhcp/10.10.54.1 with 409 (1.33 ms)

and my way to check what is in the DHCP reservations has been to run

#curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem https://foreman-1.app.domain:9090/dhcp/10.10.54.0 | json_reformat

{
    "reservations": [
        {
            "name": "centos.app.bastet11.net",
            "ip": "10.10.54.223",
            "mac": "56:6f:c4:a5:00:07",
            "subnet": "10.10.54.0/255.255.255.0",
            "type": "reservation",
            "deleteable": true,
            "hostname": "centos.app.bastet11.net",
            "hardware_type": "ethernet",
            "filename": "grub2/grubx64.efi",
            "nextServer": "10.10.54.252"
        }
    ],
    "leases": [
        {
         ...
        }
    ]
}

There are a bunch of leases in there that I’m not worried about.

I also deleted the host from the UI to clear everything out and when I curl the DHCP API, I see see the reservation even though the log says it was remove.
proxy.log when deleting the host from foreman UI

2021-10-14T23:12:00 d9c82825 [I] Started DELETE /dns/centos.app.bastet11.net/A 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /dns/centos.app.bastet11.net/A with 200 (351.68 ms)
2021-10-14T23:12:00 d9c82825 [I] Started DELETE /dns/223.54.10.10.in-addr.arpa 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /dns/223.54.10.10.in-addr.arpa with 200 (216.31 ms)
2021-10-14T23:12:00 d9c82825 [I] Started GET /tftp/serverName 
2021-10-14T23:12:00 d9c82825 [I] Finished GET /tftp/serverName with 200 (0.57 ms)
2021-10-14T23:12:00 d9c82825 [I] Started DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 200 (0.44 ms)
2021-10-14T23:12:00 d9c82825 [I] Started DELETE /tftp/PXELinux/56:6f:c4:a5:00:07 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /tftp/PXELinux/56:6f:c4:a5:00:07 with 200 (1.18 ms)
2021-10-14T23:12:00 d9c82825 [I] Started DELETE /tftp/PXEGrub2/56:6f:c4:a5:00:07 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /tftp/PXEGrub2/56:6f:c4:a5:00:07 with 200 (0.71 ms)
2021-10-14T23:12:00 d9c82825 [I] Started DELETE /tftp/PXEGrub/56:6f:c4:a5:00:07 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /tftp/PXEGrub/56:6f:c4:a5:00:07 with 200 (1.95 ms)
2021-10-14T23:12:01 d9c82825 [I] Started DELETE /tftp/iPXE/56:6f:c4:a5:00:07 
2021-10-14T23:12:01 d9c82825 [I] Finished DELETE /tftp/iPXE/56:6f:c4:a5:00:07 with 200 (0.63 ms)
2021-10-14T23:12:01 d9c82825 [I] Started DELETE /realm/BASTET11.NET/centos.app.bastet11.net 
2021-10-14T23:12:01 d9c82825 [I] Finished DELETE /realm/BASTET11.NET/centos.app.bastet11.net with 200 (515.61 ms)
2021-10-14T23:12:01 d9c82825 [I] Started DELETE /puppet/ca/autosign/centos.app.bastet11.net 
2021-10-14T23:12:01 d9c82825 [E] Attempt to remove nonexistent client autosign for centos.app.bastet11.net
2021-10-14T23:12:01 d9c82825 [W] Error details for Attempt to remove nonexistent client autosign for centos.app.bastet11.net: <Exception>: Attempt to remove nonexistent client autosign for centos.app.bastet11.net
2021-10-14T23:12:01 d9c82825 [W] Attempt to remove nonexistent client autosign for centos.app.bastet11.net: <Exception>: Attempt to remove nonexistent client autosign for centos.app.bastet11.net
2021-10-14T23:12:01 d9c82825 [I] Finished DELETE /puppet/ca/autosign/centos.app.bastet11.net with 404 (2.03 ms)
2021-10-14T23:12:01 d9c82825 [I] Started DELETE /puppet/ca/centos.app.bastet11.net 
2021-10-14T23:12:02 d9c82825 [I] Finished DELETE /puppet/ca/centos.app.bastet11.net with 200 (395.54 ms)
2021-10-14T23:13:03 90a5f3a9 [I] Started GET /dhcp/10.10.54.0 
2021-10-14T23:13:03 90a5f3a9 [I] Finished GET /dhcp/10.10.54.0 with 200 (0.9 ms)
2021-10-14T23:14:00 7a4505f5 [I] Started GET /dhcp/10.10.54.0 
2021-10-14T23:14:00 7a4505f5 [I] Finished GET /dhcp/10.10.54.0 with 200 (0.62 ms)

Tell me what IP address the host is discovered with?

And what IP address it ends up with?

What would be great to see dhcpd.leases after discovery and then after provisioning. You can delete irrelevant entries, just those which matter (same IP or MAC).