DHCP entries never deleted

Support
,
1

Problem:
I’m rebuidling my foreman instance and running into problems with the built in DHCP server. I have it setup fine to handle discovery and initial provisioning, but if I try to reprovision the server, I get a DHCP conflict. This also happens if I delete a server and rediscover it, the DHCP entries are never cleaned up. I havent seen any other posts talking about a similar issue, so I’m hoping I just missed a setting and will apprecieate any help.

My work around is to manually delete the DHCP entry through the API

curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem -X "DELETE" https://foreman-1.app.domain:9090/dhcp/10.10.54.0/ip/10.10.54.25

After manually deleting the DHCP entry, I can rebuild the host without issues.

Thanks!

Expected outcome:
I expect foreman to overwrite DHCP entries it manages.

Foreman and Proxy versions:
3.0.0

Foreman and Proxy plugin versions:
foreman-tasks: 5.1.0
foreman_ansible: 6.4.1
foreman_bootdisk: 18.0.0
foreman_discovery: 18.0.0
foreman_puppet: 1.0.3
foreman_remote_execution: 4.7.0
katello: 4.2.0.1.rc3

Distribution and version:
Red Hat Enterprise Linux release 8.4 (Ootpa)

Other relevant data:
Configuration:

foreman-installer \
--foreman-proxy-dhcp=true \
--foreman-proxy-dhcp-gateway=10.10.54.1 \
--foreman-proxy-dhcp-range="10.10.54.10 10.10.54.249" \
--foreman-proxy-dhcp-nameservers="10.10.54.250,10.10.54.251" \
--foreman-proxy-tftp=true \
--enable-foreman-plugin-discovery \
--enable-foreman-proxy-plugin-discovery \
--foreman-proxy-plugin-discovery-install-images=true \
--foreman-proxy-http true \
--foreman-proxy-httpboot=true \
--enable-foreman-plugin-remote-execution \
--enable-foreman-proxy-plugin-remote-execution-ssh \
--enable-foreman-plugin-ansible --enable-foreman-proxy-plugin-ansible \
--enable-foreman-plugin-bootdisk

UI subnet config:

Screenshot from 2021-10-13 23-05-29
Screenshot from 2021-10-13 23-05-29761×902 36.8 KB

In the UI, I receive this message

Failed to enable centos.app.domain for installation: ["Create DHCP Settings for centos.app.domain task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://foreman-1.app.domain:9090/dhcp", "Failed to perform rollback on Remove DHCP Settings for centos.app.domain - ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://foreman-1.app.domain:9090/dhcp"]

/var/log/foreman-proxy/proxy.log

2021-10-13T22:47:28 43dc6537 [I] Started GET /v2/features 
2021-10-13T22:47:28 43dc6537 [I] Finished GET /v2/features with 200 (226.26 ms)
2021-10-13T22:47:32 f5fa75db [I] Started GET /tftp/serverName 
2021-10-13T22:47:32 f5fa75db [I] Finished GET /tftp/serverName with 200 (0.67 ms)
2021-10-13T22:47:32 f5fa75db [I] Started GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-13T22:47:32 f5fa75db [E] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [W] Error details for No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [W] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [I] Finished GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 404 (1.71 ms)
2021-10-13T22:47:32 f5fa75db [I] Started GET /dhcp/10.10.54.1/ip/10.10.54.233 
2021-10-13T22:47:32 f5fa75db [E] No DHCP records for IP 10.10.54.1/10.10.54.233 found
2021-10-13T22:47:32 f5fa75db [W] Error details for No DHCP records for IP 10.10.54.1/10.10.54.233 found: <Exception>: No DHCP records for IP 10.10.54.1/10.10.54.233 found
2021-10-13T22:47:32 f5fa75db [W] No DHCP records for IP 10.10.54.1/10.10.54.233 found: <Exception>: No DHCP records for IP 10.10.54.1/10.10.54.233 found
2021-10-13T22:47:32 f5fa75db [I] Finished GET /dhcp/10.10.54.1/ip/10.10.54.233 with 404 (0.81 ms)
2021-10-13T22:47:32 f5fa75db [I] Started GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-13T22:47:32 f5fa75db [E] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [W] Error details for No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [W] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-13T22:47:32 f5fa75db [I] Finished GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 404 (0.69 ms)
2021-10-13T22:47:32 f5fa75db [I] Started GET /tftp/serverName 
2021-10-13T22:47:32 f5fa75db [I] Finished GET /tftp/serverName with 200 (0.56 ms)
2021-10-13T22:47:32 f5fa75db [I] Started DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-13T22:47:32 f5fa75db [I] Finished DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 200 (0.53 ms)
2021-10-13T22:47:32 f5fa75db [I] Started POST /dhcp/10.10.54.1 
2021-10-13T22:47:32 f5fa75db [W] Request to create a conflicting DHCP record
2021-10-13T22:47:32 f5fa75db [E] Record 10.10.54.0/10.10.54.233 already exists
2021-10-13T22:47:32 f5fa75db [W] Error details for Record 10.10.54.0/10.10.54.233 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.233 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-13T22:47:32 f5fa75db [W] Record 10.10.54.0/10.10.54.233 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.233 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-13T22:47:32 f5fa75db [I] Finished POST /dhcp/10.10.54.1 with 409 (4.08 ms)
2021-10-13T22:47:32 f5fa75db [I] Started POST /dhcp/10.10.54.1 
2021-10-13T22:47:32 f5fa75db [W] Request to create a conflicting DHCP record
2021-10-13T22:47:32 f5fa75db [E] Record 10.10.54.0/10.10.54.233 already exists
2021-10-13T22:47:32 f5fa75db [W] Error details for Record 10.10.54.0/10.10.54.233 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.233 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-13T22:47:32 f5fa75db [W] Record 10.10.54.0/10.10.54.233 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.233 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-13T22:47:32 f5fa75db [I] Finished POST /dhcp/10.10.54.1 with 409 (1.45 ms)
2

Set the DHCP pool range as 10-99 and from/to IP range as 100-250. No intersections.

When you discover a host it will get a lease from 10-99, when you provision it it will query our DHCP module to find a free IP in the 100-250 range.

For the record, dhcpd.leases is a append only file, things are deleted there after server performs a squash. Search for the deleted keyword.

3

Thanks Izap for the configuration change. I just tried it out and I’m still getting the DHCP conflicts

I ran this installer command

foreman-installer  --foreman-proxy-dhcp-range="10.10.54.10 10.10.54.99"

and then updated my app subnet to the 100-249 range

Screenshot from 2021-10-14 23-05-39
Screenshot from 2021-10-14 23-05-39744×887 36.3 KB

I can use discovery to boot the server, do the initial provision without a problem. If I try to provision an already configured host, I still get the same error in the UI:

Failed to enable centos.app.domain for installation: ["Create DHCP Settings for centos.app.domain task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://foreman-1.app.domain:9090/dhcp", "Failed to perform rollback on Remove DHCP Settings for centos.app.domain - ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Conflict]: 409 Conflict) for proxy https://foreman-1.app.domain:9090/dhcp"]

Here’s proxy.log when I run the build command

2021-10-14T23:02:38 b46cdace [I] Started GET /v2/features 
2021-10-14T23:02:38 b46cdace [I] Finished GET /v2/features with 200 (19.0 ms)
2021-10-14T23:02:39 eca0104a [I] Started GET /tftp/serverName 
2021-10-14T23:02:39 eca0104a [I] Finished GET /tftp/serverName with 200 (0.5 ms)
2021-10-14T23:02:39 eca0104a [I] Started GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-14T23:02:39 eca0104a [E] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [W] Error details for No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [W] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [I] Finished GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 404 (1.34 ms)
2021-10-14T23:02:39 eca0104a [I] Started GET /dhcp/10.10.54.1/ip/10.10.54.223 
2021-10-14T23:02:39 eca0104a [E] No DHCP records for IP 10.10.54.1/10.10.54.223 found
2021-10-14T23:02:39 eca0104a [W] Error details for No DHCP records for IP 10.10.54.1/10.10.54.223 found: <Exception>: No DHCP records for IP 10.10.54.1/10.10.54.223 found
2021-10-14T23:02:39 eca0104a [W] No DHCP records for IP 10.10.54.1/10.10.54.223 found: <Exception>: No DHCP records for IP 10.10.54.1/10.10.54.223 found
2021-10-14T23:02:39 eca0104a [I] Finished GET /dhcp/10.10.54.1/ip/10.10.54.223 with 404 (1.15 ms)
2021-10-14T23:02:39 eca0104a [I] Started GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-14T23:02:39 eca0104a [E] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [W] Error details for No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [W] No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found: <Exception>: No DHCP record for MAC 10.10.54.1/56:6f:c4:a5:00:07 found
2021-10-14T23:02:39 eca0104a [I] Finished GET /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 404 (0.73 ms)
2021-10-14T23:02:39 eca0104a [I] Started GET /tftp/serverName 
2021-10-14T23:02:39 eca0104a [I] Finished GET /tftp/serverName with 200 (0.56 ms)
2021-10-14T23:02:39 eca0104a [I] Started DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-14T23:02:39 eca0104a [I] Finished DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 200 (0.54 ms)
2021-10-14T23:02:39 eca0104a [I] Started POST /dhcp/10.10.54.1 
2021-10-14T23:02:39 eca0104a [W] Request to create a conflicting DHCP record
2021-10-14T23:02:39 eca0104a [E] Record 10.10.54.0/10.10.54.223 already exists
2021-10-14T23:02:39 eca0104a [W] Error details for Record 10.10.54.0/10.10.54.223 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.223 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-14T23:02:39 eca0104a [W] Record 10.10.54.0/10.10.54.223 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.223 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-14T23:02:39 eca0104a [I] Finished POST /dhcp/10.10.54.1 with 409 (4.31 ms)
2021-10-14T23:02:39 eca0104a [I] Started POST /dhcp/10.10.54.1 
2021-10-14T23:02:39 eca0104a [W] Request to create a conflicting DHCP record
2021-10-14T23:02:39 eca0104a [E] Record 10.10.54.0/10.10.54.223 already exists
2021-10-14T23:02:39 eca0104a [W] Error details for Record 10.10.54.0/10.10.54.223 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.223 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-14T23:02:39 eca0104a [W] Record 10.10.54.0/10.10.54.223 already exists: <Proxy::DHCP::Collision>: Record 10.10.54.0/10.10.54.223 already exists
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:157:in `add_record'
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:29:in `add_record'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:87:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2021-10-14T23:02:39 eca0104a [I] Finished POST /dhcp/10.10.54.1 with 409 (1.33 ms)

and my way to check what is in the DHCP reservations has been to run

#curl --cert /etc/foreman/client_cert.pem --key /etc/foreman/client_key.pem --cacert /etc/foreman/proxy_ca.pem https://foreman-1.app.domain:9090/dhcp/10.10.54.0 | json_reformat

{
    "reservations": [
        {
            "name": "centos.app.bastet11.net",
            "ip": "10.10.54.223",
            "mac": "56:6f:c4:a5:00:07",
            "subnet": "10.10.54.0/255.255.255.0",
            "type": "reservation",
            "deleteable": true,
            "hostname": "centos.app.bastet11.net",
            "hardware_type": "ethernet",
            "filename": "grub2/grubx64.efi",
            "nextServer": "10.10.54.252"
        }
    ],
    "leases": [
        {
         ...
        }
    ]
}

There are a bunch of leases in there that I’m not worried about.

I also deleted the host from the UI to clear everything out and when I curl the DHCP API, I see see the reservation even though the log says it was remove.
proxy.log when deleting the host from foreman UI

2021-10-14T23:12:00 d9c82825 [I] Started DELETE /dns/centos.app.bastet11.net/A 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /dns/centos.app.bastet11.net/A with 200 (351.68 ms)
2021-10-14T23:12:00 d9c82825 [I] Started DELETE /dns/223.54.10.10.in-addr.arpa 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /dns/223.54.10.10.in-addr.arpa with 200 (216.31 ms)
2021-10-14T23:12:00 d9c82825 [I] Started GET /tftp/serverName 
2021-10-14T23:12:00 d9c82825 [I] Finished GET /tftp/serverName with 200 (0.57 ms)
2021-10-14T23:12:00 d9c82825 [I] Started DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /dhcp/10.10.54.1/mac/56:6f:c4:a5:00:07 with 200 (0.44 ms)
2021-10-14T23:12:00 d9c82825 [I] Started DELETE /tftp/PXELinux/56:6f:c4:a5:00:07 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /tftp/PXELinux/56:6f:c4:a5:00:07 with 200 (1.18 ms)
2021-10-14T23:12:00 d9c82825 [I] Started DELETE /tftp/PXEGrub2/56:6f:c4:a5:00:07 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /tftp/PXEGrub2/56:6f:c4:a5:00:07 with 200 (0.71 ms)
2021-10-14T23:12:00 d9c82825 [I] Started DELETE /tftp/PXEGrub/56:6f:c4:a5:00:07 
2021-10-14T23:12:00 d9c82825 [I] Finished DELETE /tftp/PXEGrub/56:6f:c4:a5:00:07 with 200 (1.95 ms)
2021-10-14T23:12:01 d9c82825 [I] Started DELETE /tftp/iPXE/56:6f:c4:a5:00:07 
2021-10-14T23:12:01 d9c82825 [I] Finished DELETE /tftp/iPXE/56:6f:c4:a5:00:07 with 200 (0.63 ms)
2021-10-14T23:12:01 d9c82825 [I] Started DELETE /realm/BASTET11.NET/centos.app.bastet11.net 
2021-10-14T23:12:01 d9c82825 [I] Finished DELETE /realm/BASTET11.NET/centos.app.bastet11.net with 200 (515.61 ms)
2021-10-14T23:12:01 d9c82825 [I] Started DELETE /puppet/ca/autosign/centos.app.bastet11.net 
2021-10-14T23:12:01 d9c82825 [E] Attempt to remove nonexistent client autosign for centos.app.bastet11.net
2021-10-14T23:12:01 d9c82825 [W] Error details for Attempt to remove nonexistent client autosign for centos.app.bastet11.net: <Exception>: Attempt to remove nonexistent client autosign for centos.app.bastet11.net
2021-10-14T23:12:01 d9c82825 [W] Attempt to remove nonexistent client autosign for centos.app.bastet11.net: <Exception>: Attempt to remove nonexistent client autosign for centos.app.bastet11.net
2021-10-14T23:12:01 d9c82825 [I] Finished DELETE /puppet/ca/autosign/centos.app.bastet11.net with 404 (2.03 ms)
2021-10-14T23:12:01 d9c82825 [I] Started DELETE /puppet/ca/centos.app.bastet11.net 
2021-10-14T23:12:02 d9c82825 [I] Finished DELETE /puppet/ca/centos.app.bastet11.net with 200 (395.54 ms)
2021-10-14T23:13:03 90a5f3a9 [I] Started GET /dhcp/10.10.54.0 
2021-10-14T23:13:03 90a5f3a9 [I] Finished GET /dhcp/10.10.54.0 with 200 (0.9 ms)
2021-10-14T23:14:00 7a4505f5 [I] Started GET /dhcp/10.10.54.0 
2021-10-14T23:14:00 7a4505f5 [I] Finished GET /dhcp/10.10.54.0 with 200 (0.62 ms)
4

Tell me what IP address the host is discovered with?

And what IP address it ends up with?

What would be great to see dhcpd.leases after discovery and then after provisioning. You can delete irrelevant entries, just those which matter (same IP or MAC).

5

OK, here’s the dhcpd.leases for just the selected mac address file after discovery, the VM was assigned 10.10.54.85

lease 10.10.54.71 {
  starts 6 2021/10/16 00:56:50;
  ends 6 2021/10/16 12:56:50;
  cltt 6 2021/10/16 00:56:50;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 56:6f:c4:a5:00:0a;
  uid "\000\271s\347\226d\270\345N\222\006\036\263\371\273\362.";
  set vendor-class-identifier = "PXEClient:Arch:00007:UNDI:003001";
  set vendor-string = "PXEClient:Arch:00007:UNDI:003001";
}
lease 10.10.54.85 {
  starts 6 2021/10/16 00:58:24;
  ends 6 2021/10/16 12:58:24;
  cltt 6 2021/10/16 00:58:24;
  binding state active;
  next binding state free;
  rewind binding state free;
  hardware ethernet 56:6f:c4:a5:00:0a;
}

After provisioning, it was assigned 10.10.54.181. Here’s the dhcpd.leases file in that case.

host centos-test.app.domain {
  dynamic;
  hardware ethernet 56:6f:c4:a5:00:0a;
  fixed-address 10.10.54.181;
        supersede server.filename = "grub2/grubx64.efi";
        supersede server.next-server = 0a:0a:36:fc;
        supersede host-name = "centos-test.app.domain";
}
lease 10.10.54.71 {
  starts 6 2021/10/16 00:56:50;
  ends 6 2021/10/16 01:02:20;
  tstp 6 2021/10/16 01:02:20;
  cltt 6 2021/10/16 00:56:50;
  binding state free;
  hardware ethernet 56:6f:c4:a5:00:0a;
  uid "\000\271s\347\226d\270\345N\222\006\036\263\371\273\362.";
}
lease 10.10.54.85 {
  starts 6 2021/10/16 00:58:24;
  ends 6 2021/10/16 12:58:24;
  tstp 6 2021/10/16 12:58:24;
  cltt 6 2021/10/16 00:58:24;
  binding state free;
  hardware ethernet 56:6f:c4:a5:00:0a;
}

At this point, if I try hit build on the host page to reinstall from scratch, I get the DHCP conflict message referenced initially. I have to delete the host entry through the API as initially mentioned to get it to move. After deleting the host entry, it recreates the host entry for PXE install and installs fine again.

host centos-test.app.domain {
  dynamic;
  hardware ethernet 56:6f:c4:a5:00:0a;
  fixed-address 10.10.54.181;
        supersede server.filename = "grub2/grubx64.efi";
        supersede server.next-server = 0a:0a:36:fc;
        supersede host-name = "centos-test.app.domain";
}
6

Sorry for the delay. I cannot reproduce. For the last example, what was the conflict error?

So my understanding is you have an existing (provisioned) host with a working reservation. Then you click on Build button in the Foreman UI and it fails with a conflict?

If a newly provisioned host is provisioned with address and nothing change it (e.g. operator or facts coming out from puppet/ansible/rhsm), no DHCP will be done. There is no need to update reservation, if you are booting from network Foreman will just change PXE bootloader configuration.

7

Yes, I get a conflict error in this case, after it was provisioned and foreman created the reservation It works fine as long as there is no existing reservation, aka when I create a new VM and use discovery to provision.

You are correct, I get the DHCP conflict message when attempting to click build on an already provisioned server.

This might not be a typical use case, but I’m trying to work on application install scripts so I can document what I’m doing and hopefully automate the application installs. I try to start over and provision the VM over itself to make sure my application installs are getting better with each pass. Instead of deleting the VM and creating a new one, I’ve just been using the same VM and manually deleting (through the API) the reservation for foreman to provision again.

Since you cant reproduce, it must be some kind of config problem on with my install, though I tried to leave defaults as much as possible. I can use curl to manually delete the reservation through the foreman-proxy API, so I dont think that’s the problem. Is there any other services I should be looking at logging to see if I can track down what’s going on?

8

The most important question: Why Foreman wants to orchestrate a DHCP change when you hit the Build button?

If you provision a server with IPv4 X, then you can rebuild it many times and as long as the IP address of the managed interface is still X, Foreman will not do any DHCP calls.

Investigate what IP addresses your server is provisioned with and then rebuilt with.