Can't create new host, fails at DHCP or TFTP?

Problem: Utilizing the webGUI to create a new host fails at setting DHCP settings… but logs also show that in.tftpd doesn’t like some options… but I’m not sure what options they are talking about.

Expected outcome: Host successfully creates, allowing a kickstart/PXE boot of a new system

Foreman and Proxy versions: 3.7

Distribution and version: Ubuntu 20.04 LTS

Other relevant data:
From /var/log/foreman/production.log:

2023-11-08T13:12:16 [W|app|6737cc69] Rolling back due to a problem: [#<Orchestration::Task:0x00005587d2253280 @name="Create DHCP Settings for cobblertest.spectric.dev", @id="dhcp_create_00:0c:29:43:12:d5", @status="failed", @priority=10, @action=[#<Nic::Managed id: nil, mac: "00:0c:29:43:12:d5", ip: "10.48.30.66", type: "Nic::Managed", name: "cobblertest.spectric.dev", host_id: nil, subnet_id: 1, domain_id: 2, attrs: {}, created_at: nil, updated_at: nil, provider: nil, username: nil, password: nil, virtual: false, link: true, identifier: "ens160", tag: "", attached_to: "", managed: true, mode: "balance-rr", attached_devices: "", bond_options: "", primary: true, provision: true, compute_attributes: {}, ip6: "", subnet6_id: nil>, :set_dhcp], @created=1699449076.4060335, @timestamp=2023-11-08 13:12:16.567874629 UTC>]
2023-11-08T13:12:16 [E|app|6737cc69] Failed to save: Create DHCP Settings for cobblertest.spectric.dev task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::Exceptions::ReadTimeout]: Timed out reading data from server) for proxy https://puppet.spectric.dev:8443/dhcp

Then /var/log/foreman-proxy/proxy.log shows:

2023-11-08T13:07:41  [I] WEBrick::HTTPServer#start: pid=167619 port=8443
2023-11-08T13:07:41  [I] Smart proxy has launched on 1 socket(s), waiting for requests
2023-11-08T13:10:55 d9d2e840 [I] Started GET /dhcp/10.48.30.0/unused_ip from=10.48.30.51&to=10.48.30.100
2023-11-08T13:10:57 d9d2e840 [I] Finished GET /dhcp/10.48.30.0/unused_ip with 200 (2013.09 ms)
2023-11-08T13:11:05 17066cb0 [I] Started GET /dhcp/10.48.30.0/unused_ip mac=00:0c:29:43:12:d5&from=10.48.30.51&to=10.48.30.100
2023-11-08T13:11:07 17066cb0 [I] Finished GET /dhcp/10.48.30.0/unused_ip with 200 (2004.36 ms)
2023-11-08T13:11:16 6737cc69 [I] Started GET /tftp/serverName
2023-11-08T13:11:16 6737cc69 [I] Finished GET /tftp/serverName with 200 (0.72 ms)
2023-11-08T13:11:16 6737cc69 [I] Started GET /tftp/serverName
2023-11-08T13:11:16 6737cc69 [I] Finished GET /tftp/serverName with 200 (0.55 ms)
2023-11-08T13:11:16 6737cc69 [I] Started GET /dhcp/10.48.30.0/mac/00:0c:29:43:12:d5
2023-11-08T13:11:16 6737cc69 [E] No DHCP record for MAC 10.48.30.0/00:0c:29:43:12:d5 found
2023-11-08T13:11:16 6737cc69 [W] Error details for No DHCP record for MAC 10.48.30.0/00:0c:29:43:12:d5 found: <Exception>: No DHCP record for MAC 10.48.30.0/00:0c:29:43:12:d5 found
2023-11-08T13:11:16 6737cc69 [W] No DHCP record for MAC 10.48.30.0/00:0c:29:43:12:d5 found: <Exception>: No DHCP record for MAC 10.48.30.0/00:0c:29:43:12:d5 found
2023-11-08T13:11:16 6737cc69 [I] Finished GET /dhcp/10.48.30.0/mac/00:0c:29:43:12:d5 with 404 (4.13 ms)
2023-11-08T13:11:16 6737cc69 [I] Started GET /dhcp/10.48.30.0/ip/10.48.30.66
2023-11-08T13:11:16 6737cc69 [E] No DHCP records for IP 10.48.30.0/10.48.30.66 found
2023-11-08T13:11:16 6737cc69 [W] Error details for No DHCP records for IP 10.48.30.0/10.48.30.66 found: <Exception>: No DHCP records for IP 10.48.30.0/10.48.30.66 found
2023-11-08T13:11:16 6737cc69 [W] No DHCP records for IP 10.48.30.0/10.48.30.66 found: <Exception>: No DHCP records for IP 10.48.30.0/10.48.30.66 found
2023-11-08T13:11:16 6737cc69 [I] Finished GET /dhcp/10.48.30.0/ip/10.48.30.66 with 404 (2.42 ms)
2023-11-08T13:11:16 6737cc69 [I] Started POST /dhcp/10.48.30.0

I think I’m missing something simple but I can’t figure it out?

To me it looks like it’s looking at a server with IP 10.48.30.0 when it should be looking at 10.48.30.102 (where DHCP is managed) … then it would see the dhcp.leases exists and has an entry for it, no? Wonder where I can change that …

Only entry I found in anything foreman related is in settings.d/dhcp.yml … but that’s just part of the subnet… then, when I change the IP address of the DHCP server to the foreman host (it was pointing to our DNS server, which also does have DHCP on it… but DHCP is running on the local foreman host) it fails even before that stating 400 bad request, can’t set the DHCP entry.

Re-ran foreman-installer with a few options changed and it created… so, solved?