DHCP lease creation failing silently

Problem:

Foreman is not creating DHCP leases using the isc_dhcp plugin. This works with our old instance to the same DHCP server, but not on Foreman 2.3.1/Katello 3.18.1

Expected outcome:

Leases would be created

Foreman and Proxy versions:

foreman-proxy-2.3.1-1.el8.noarch

Distribution and version:

CentOS 8

Other relevant data:

In the logs, I see that the proxy thinks it has created a lease:

2021-02-04T18:04:18 c2619cce [I] Started POST /dhcp/192.168.1.0
2021-02-04T18:04:18 c2619cce [D] verifying remote client 192.168.1.192 against trusted_hosts ["u14man01v.dev.identrust.com"]
2021-02-04T18:04:18 c2619cce [D] omshell> key [filtered] [filtered]
2021-02-04T18:04:18 c2619cce [D] omshell> server 192.168.1.30
2021-02-04T18:04:18 c2619cce [D] omshell> port 7911
2021-02-04T18:04:18 c2619cce [D] omshell> connect
2021-02-04T18:04:18 c2619cce [D] omshell> new host
2021-02-04T18:04:18 c2619cce [D] omshell> set name = "earl-dulan.dev.identrust.com"
2021-02-04T18:04:18 c2619cce [D] omshell> set ip-address = 192.168.1.153
2021-02-04T18:04:18 c2619cce [D] omshell> set hardware-address = 00:50:56:af:c0:12
2021-02-04T18:04:18 c2619cce [D] omshell> set hardware-type = 1
2021-02-04T18:04:18 c2619cce [D] omshell> set statements = "filename = \"pxelinux.0\"; option host-name = \"earl-dulan.dev.identrust.com\";"
2021-02-04T18:04:18 c2619cce [D] omshell> create
2021-02-04T18:04:18 c2619cce [D] Added DHCP reservation for earl-dulan.dev.identrust.com (10.3.1.153 / 00:50:56:af:c0:12)
2021-02-04T18:04:18 c2619cce [I] Finished POST /dhcp/192.168.1.0 with 200 (11.08 ms)
2021-02-04T18:04:19  [D] close: 192.168.1.192:42010
2021-02-04T18:04:19  [D] accept: 192.168.1.192:42012
2021-02-04T18:04:19  [D] Rack::Handler::WEBrick is invoked.

But when I run a packet capture using TCP dump, there is no traffic to the DHCP server on the omapi port:

[jshewey@foreman01 ~]$ sudo tcpdump -s 0 -i any -w /home/7911.pcap host 192.168.1.30
dropped privs to tcpdump
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
^C0 packets captured
0 packets received by filter
0 packets dropped by kernel

Config files:

[jshewey@u14man01v ~]$ cat /etc/foreman-proxy/settings.d/dhcp.yml
---
# Enable DHCP management
# Can be true, false, or http/https to enable just one of the protocols
:enabled: true

# valid providers:
#   - dhcp_isc (ISC dhcp server)
#   - dhcp_native_ms (Microsoft native implementation)
#   - dhcp_libvirt (dnsmasq via libvirt)
:use_provider: dhcp_isc
:server: 10.3.1.30
# subnets restricts the subnets queried to a subset, to reduce the query time.
:subnets:
#  - 192.168.205.0/255.255.255.128
#  - 192.168.205.128/255.255.255.128
  - 192.168.1.0/255.255.255.0
[jshewey@u14man01v ~]$ cat /etc/foreman-proxy/settings.d/dhcp_isc.yml
---
#
# Configuration file for ISC dhcp provider
#

:config: /etc/dhcp/dhcpd.conf
:leases: /var/lib/dhcpd/dhcpd.leases

# Redhat 5
#
#:config: /etc/dhcpd.conf
#
# Settings for Ubuntu
#
#:config: /etc/dhcp3/dhcpd.conf
#:leases: /var/lib/dhcp3/dhcpd.leases

# Specifies TSIG key name and secret

:key_name: omapi_key
:key_secret: [redacted]

:omapi_port: 7911

# use :server setting in dhcp.yml if you are managing a dhcp server which is not localhost

My educated guess is that SELinux is blocking DHCP daemon from connecting there.

selinux is disable on both hosts.

The solution ended up being that I had to change :enabled: true to :enabled: https in /etc/foreman-proxy/settings.d/dhcp.yml