Xyz is static - unable to delete

Problem:
While trying to “build” a host that is going to be deployed via PXE 8 (from a proxy), I get the

2022-07-06T15:19:18 d6244e58 [W] 10.38.180.6 / 00:50:56:87:09:c8 is static - unable to delete: <Proxy::DHCP::InvalidRecord>: 10.38.180.6 / 00:50:56:87:09:c8 is static - unable to delete

message in the proxy log

also referenced here:

Foreman and Proxy versions:

foreman-proxy-3.2.1-1.el8.noarch

Foreman and Proxy plugin versions:

Distribution and version:

CentOS 8 Stream
Linux ew-pxe02-prod.ewsrv.ch 4.18.0-394.el8.x86_64 #1 SMP Tue May 31 16:19:11 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

Other relevant data:

2022-07-06T15:19:18 d6244e58 [I] Started DELETE /dhcp/10.38.0.0/mac/00:50:56:87:09:c8 
2022-07-06T15:19:18 d6244e58 [E] 10.38.180.6 / 00:50:56:87:09:c8 is static - unable to delete
2022-07-06T15:19:18 d6244e58 [W] Error details for 10.38.180.6 / 00:50:56:87:09:c8 is static - unable to delete: <Proxy::DHCP::InvalidRecord>: 10.38.180.6 / 00:50:56:87:09:c8 is static - unable to delete
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:25:in `del_record'
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:93:in `del_record_by_mac'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:136:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-07-06T15:19:18 d6244e58 [W] 10.38.180.6 / 00:50:56:87:09:c8 is static - unable to delete: <Proxy::DHCP::InvalidRecord>: 10.38.180.6 / 00:50:56:87:09:c8 is static - unable to delete
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:25:in `del_record'
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:93:in `del_record_by_mac'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:136:in `block in <class:DhcpApi>'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1675:in `block in compile!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (3 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1032:in `route_eval'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1013:in `block (2 levels) in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1061:in `block in process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1059:in `process_route'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1011:in `block in route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `each'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1008:in `route!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1129:in `block in dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1124:in `dispatch!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `block in call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `block in invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `catch'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1101:in `invoke'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:939:in `call!'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:929:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call'
/usr/share/foreman-proxy/lib/proxy/log.rb:105:in `call'
/usr/share/foreman-proxy/lib/proxy/request_id_middleware.rb:11:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/xss_header.rb:18:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/path_traversal.rb:16:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/json_csrf.rb:26:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/base.rb:50:in `call'
/usr/share/gems/gems/rack-protection-2.1.0/lib/rack/protection/frame_options.rb:31:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/null_logger.rb:11:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/show_exceptions.rb:22:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:216:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1991:in `call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `block in call'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1769:in `synchronize'
/usr/share/gems/gems/sinatra-2.1.0/lib/sinatra/base.rb:1542:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each'
/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/builder.rb:244:in `call'
/usr/share/gems/gems/rack-2.2.3/lib/rack/handler/webrick.rb:95:in `service'
/usr/share/ruby/webrick/httpserver.rb:140:in `service'
/usr/share/ruby/webrick/httpserver.rb:96:in `run'
/usr/share/ruby/webrick/server.rb:307:in `block in start_thread'
/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'
2022-07-06T15:19:18 d6244e58 [I] Finished DELETE /dhcp/10.38.0.0/mac/00:50:56:87:09:c8 with 400 (1.13 ms)

Can I just delete the lease manually from /var/lib/dhcpd/dhcpd.leases?

So I cleared the dhcpd.leases file, but I still get:

2022-07-09T00:16:21 a8883ef1 [I] Started GET /dhcp/10.38.0.0/mac/00:50:56:87:09:c8 
2022-07-09T00:16:21 a8883ef1 [I] Finished GET /dhcp/10.38.0.0/mac/00:50:56:87:09:c8 with 200 (0.71 ms)
2022-07-09T00:16:22 934c8e69 [I] Started GET /unattended/templateServer 
2022-07-09T00:16:22 934c8e69 [I] Finished GET /unattended/templateServer with 200 (0.39 ms)
2022-07-09T00:16:26 a8883ef1 [I] Started GET /dhcp/10.38.0.0/ip/10.38.185.249 
2022-07-09T00:16:26 a8883ef1 [E] No DHCP records for IP 10.38.0.0/10.38.185.249 found
2022-07-09T00:16:26 a8883ef1 [W] Error details for No DHCP records for IP 10.38.0.0/10.38.185.249 found: <Exception>: No DHCP records for IP 10.38.0.0/10.38.185.249 found
2022-07-09T00:16:26 a8883ef1 [W] No DHCP records for IP 10.38.0.0/10.38.185.249 found: <Exception>: No DHCP records for IP 10.38.0.0/10.38.185.249 found
2022-07-09T00:16:26 a8883ef1 [I] Finished GET /dhcp/10.38.0.0/ip/10.38.185.249 with 404 (0.79 ms)
2022-07-09T00:16:27 934c8e69 [I] Started GET /unattended/templateServer 
2022-07-09T00:16:27 934c8e69 [I] Finished GET /unattended/templateServer with 200 (0.37 ms)
2022-07-09T00:16:31 a8883ef1 [I] Started GET /dhcp/10.38.0.0/mac/00:50:56:87:09:c8 
2022-07-09T00:16:31 a8883ef1 [I] Finished GET /dhcp/10.38.0.0/mac/00:50:56:87:09:c8 with 200 (0.66 ms)
2022-07-09T00:16:32 934c8e69 [I] Started GET /unattended/templateServer 
2022-07-09T00:16:32 934c8e69 [I] Finished GET /unattended/templateServer with 200 (0.32 ms)
2022-07-09T00:16:36 a8883ef1 [I] Started DELETE /dhcp/10.38.0.0/mac/00:50:56:87:09:c8 
2022-07-09T00:16:36 a8883ef1 [E] 10.38.180.0 / 00:50:56:87:09:c8 is static - unable to delete
2022-07-09T00:16:36 a8883ef1 [W] Error details for 10.38.180.0 / 00:50:56:87:09:c8 is static - unable to delete: <Proxy::DHCP::InvalidRecord>: 10.38.180.0 / 00:50:56:87:09:c8 is static - unable to delete
/usr/share/foreman-proxy/modules/dhcp_common/isc/omapi_provider.rb:25:in `del_record'
/usr/share/foreman-proxy/modules/dhcp_common/server.rb:93:in `del_record_by_mac'
/usr/share/foreman-proxy/modules/dhcp/dhcp_api.rb:136:in `block in <class:DhcpApi>'

What does the mean?

I tried setting IPAM for this subnet to “Internal DB”, Boot-Mode ist DHCP - but I get the same error message.

OK, so I tried without dhcp proxy (used another dhcp server), still didn’t work.

Then I found this:

and the hostgroup had no pxe-loader defined (I think on purpose, more or less).
I then set a pxe-loader and a configuration was placed in /var/lib/tftpd/pxelinx.cfg.

Mystery solved…

Now I need to check why it still doesn’t boot that configuration…