Create Reverse IPv4 DNS record for ########### task failed with the following error: ERF12-2357

Support
1

Problem:

No longer able to create hosts via Foreman Web GUI or Hammer CLI. This previously worked and has been working for several years until last week. The DHCP object doesn’t appear to be deleted.

Expected outcome:

Host created

Foreman and Proxy versions:
1.24.3

Foreman and Proxy plugin versions:

foreman-task 0.17.5
foreman-ansible 4.0.5
foreman_dhcp_browser 0.0.8
foreman_remote_execution .0.8
foreman_setup 7.0.0

Distribution and version:

[root@ukforeman1 ~]# cat /etc/os-release
NAME=“CentOS Linux”
VERSION=“7 (Core)”
ID=“centos”
ID_LIKE=“rhel fedora”
VERSION_ID=“7”
PRETTY_NAME=“CentOS Linux 7 (Core)”
ANSI_COLOR=“0;31”
CPE_NAME=“cpe:/o:centos:centos:7”
HOME_URL=“https://www.centos.org/”

Other relevant data:

I delete and create DHCP entries using omshell.

[root@ukforeman1 ~]# omshell << EOF

server 10.5.22.100
port 7911
connect
new host
set name = “my.hostname”
set ip-address = 10.5.22.78
set hardware-address = 14:b3:1f:8b:38:3b
set hardware-type = 1
create
EOF

obj:
obj: host
obj: host
name = “my.hostname”
obj: host
name = “my.hostname”
ip-address = 0a:05:16:4e
obj: host
name = “my.hostname”
ip-address = 0a:05:16:4e
hardware-address = 14:b3:1f:8b:38:3b
obj: host
name = “my.hostname”
ip-address = 0a:05:16:4e
hardware-address = 14:b3:1f:8b:38:3b
hardware-type = 1
obj: host
name = “my.hostname”
ip-address = 0a:05:16:4e
hardware-address = 14:b3:1f:8b:38:3b
hardware-type = 00:00:00:01

and appears in /var/lib/dhcpd/dhcpd.leases

host my.hostname {
dynamic;
deleted;
}
host my.hostname {
dynamic;
hardware ethernet 14:b3:1f:8b:38:3b;
fixed-address 10.5.22.78;
}

2

omshell= ip-address = 0a:05:16:4e
omshell= hardware-address = 14:b3:1f:8b:38:3b
omshell= hardware-type = 1
omshell= statements = “filename = “pxelinux.0”; next-server = “0a:05:16:4e”; option host-name = “my.hostname”;”
omshell= >
2023-06-14T14:43:07 452ab978 [E] Failed to add DHCP reservation for my.hostname (10.5.22.78 / 14:b3:1f:8b:38:3b): Entry already exists
2023-06-14T14:43:07 452ab978 [E] No DHCP record for MAC 10.5.22.0/14:b3:1f:8b:38:3b found
2023-06-14T14:43:07 452ab978 [E] No DHCP record for MAC 10.5.22.0/14:b3:1f:8b:38:3b found
2023-06-14T14:43:07 452ab978 [E] No DHCP record for MAC 10.5.22.0/14:b3:1f:8b:38:3b found
14:b3:1f:8b:38:3b2023-06-14T14:44:51 fa3f0df0 [E] No DHCP record for MAC 10.5.22.0/14:b3:1f:8b:38:3b found
2023-06-14T14:44:51 fa3f0df0 [E] No DHCP records for IP 10.5.22.0/10.5.22.78 found
2023-06-14T14:44:52 fa3f0df0 [E] ‘78.22.5.10.in-addr.arpa’ is already in use
2023-06-14T15:03:44 a19a7d8f [E] No DHCP record for MAC 10.5.22.0/14:b3:1f:8b:38:3b found
2023-06-14T15:03:44 a19a7d8f [E] No DHCP records for IP 10.5.22.0/10.5.22.78 found
2023-06-14T15:03:46 a19a7d8f [E] ‘78.22.5.10.in-addr.arpa’ is already in use

3

[root@ukforeman1 ~]# cat /etc/dhcp/dhcpd.conf

dhcpd.conf

omapi-port 7911;

default-lease-time 43200;
max-lease-time 86400;

not authoritative;

ddns-update-style none;

option domain-name “my.domain”;
option domain-name-servers 10.5.3.210, 10.2.249.210;
option ntp-servers none;

allow booting;
allow bootp;

option fqdn.no-client-update on; # set the “O” and “S” flag bits
option fqdn.rcode2 255;
option pxegrub code 150 = text ;

required for UEFI HTTP boot

if substring(option vendor-class-identifier, 0, 10) = “HTTPClient” {
option vendor-class-identifier “HTTPClient”;
}

promote vendor in dhcpd.leases

set vendor-string = option vendor-class-identifier;

next server and filename options

next-server 10.5.22.100;
option architecture code 93 = unsigned integer 16 ;
if option architecture = 00:06 {
filename “grub2/shim.efi”;
} elsif option architecture = 00:07 {
filename “grub2/shim.efi”;
} elsif option architecture = 00:09 {
filename “grub2/shim.efi”;
} else {
filename “pxelinux.0”;
}

log-facility local7;

include “/etc/dhcp/dhcpd.hosts”;

subnet 10.5.22.0 netmask 255.255.255.0 {
pool
{
range 10.5.22.150 10.5.22.254;
}

option subnet-mask 255.255.255.0;
option routers 10.5.22.1;
}

vlan20.lan

subnet 10.5.20.0 netmask 255.255.255.0 {
pool
{
range 10.5.20.101 10.5.20.254;
}

option subnet-mask 255.255.255.0;
option routers 10.5.20.1;
}
[root@ukforeman1 ~]#

4

Ansible

Version

3.0.1

DHCP

Version

1.24.3

DNS

Version

1.24.3

Domains

None found

Dynflow

Version

0.2.4

HTTPBoot

Version

1.24.3

Realm

Version

1.24.3

Realms

RMS.COM

SSH

Version

0.2.1

TFTP

Version

1.24.3

TFTP server

10.5.22.100

5

Hi and welcome to the forum :slight_smile:

First off, you are using a very old version of Foreman and Katello, which may have a lot of bugs fixed in newer version. I would highly recommend you to upgrade, since it is also hard or even impossible for people to reproduce any problems you might encounter or help with debugging.

When you encounter this error, what is the content of /var/log/foreman/production.log and /var/log/foreman-proxy/proxy.log from that timeframe? I would expect there to be more helpful messages in the log.

6

Regards
Wayne

7

Hi Areyus,

Yes, we appreciate we are running a very old version, unfortunately foreman is something I have recently inherited and resources mean updating is not an option at the moment given that it mostly does what we use it for.

----/var/log/foreman/production.log

2023-06-16T09:21:37 [E|app|40949338] Task ‘Fetch TFTP boot files for uklxfx12-cpu-05.rms.com’ canceled
2023-06-16T09:21:37 [E|app|40949338] Unprocessable entity Host::Managed (id: new):
Create DHCP Settings for uklxfx12-cpu-05.rms.com task failed with the following error: ERF12-6899 [ProxyAPI::ProxyException]: Unable to set DHCP entry ([RestClient::BadRequest]: 400 Bad Request) for proxy https://ukforeman1.rms.com:8443/dhcp

2023-06-16T09:21:37 [I|app|40949338] Rendering api/v2/errors/unprocessable_entity.json.rabl within api/v2/layouts/error_layout
2023-06-16T09:21:37 [I|app|40949338] Rendered api/v2/errors/unprocessable_entity.json.rabl within api/v2/layouts/error_layout (0.6ms)
2023-06-16T09:21:37 [I|app|40949338] Completed 422 Unprocessable Entity in 1081ms (Views: 1.2ms | ActiveRecord: 19.9ms)

----/var/log/foreman-proxy/proxy.log

omshell= statements = “filename = “pxelinux.0”; next-server = 0a:05:16:64; option host-name = “uklxfx12-cpu-05.rms.com”;”
omshell= >
2023-06-16T09:21:36 40949338 [E] Failed to add DHCP reservation for uklxfx12-cpu-05.rms.com (10.5.22.78 / 14:b3:1f:8b:38:3b): Entry already exists
2023-06-16T09:21:36 40949338 [E] No DHCP record for MAC 10.5.22.0/14:b3:1f:8b:38:3b found
2023-06-16T09:21:36 40949338 [E] No DHCP records for IP 10.5.22.0/10.5.22.78 found

Regards
Wayne

8

I am not sure I understand your workflow correctly, from what I understand you do the following;

  1. Manually add DHCP entries via omshell
  2. Create the host in Foreman
  3. Creation of the host fails due to the above error

Is this correct? The error you see is (from what I can tell in the logs) caused by step 1. Foreman tries to add a DHCP entry via omapi and finds the already existing entry. This should have never worked in the first place. If Foreman and the smart-proxy are configured correctly to handle DHCP on themselves, I would recommend just skipping step 1 and let Foreman do the management of DHCP entries by itself.
If there is a valid reason why you want to manage DHCP entries by hand though, you should remove DHCP management from the subnet(s) in Foreman.

If my assumption above is wrong and Foreman does DHCP management via omshell and you are not doing anything manually on that end: Did you delete and re-create that host and this is where the error comes from? I am not firm with how Foreman works with DHCPD since we don’t use that here. If the problem comes from an old entry not being deleted, do you happen to have the logs from the deletion still at hand?

9

Apologies, we were using the omshell to manually test and remove the host from dhcp due to a previous message relating to conflict so please ignore these messages.

We are creating the host via the hammer CLI which was working, i believe the issues appears to be this log entry

omshell= statements = “filename = “pxelinux.0”; next-server = 0a:05:16:64; option host-name = “uklxfx12-cpu-05.rms.com”;”
omshell= >
2023-06-16T09:21:36 40949338 [E] Failed to add DHCP reservation for uklxfx12-cpu-05.rms.com (10.5.22.78 / 14:b3:1f:8b:38:3b): Entry already exists

Regards
Wayne

10

Yes, this is probably what causes your problem. The underlying question remains: Where does the already existing entry come from?
My assumption was that, either:
a) Someone created it manually in DHCP before trying to create the host in Foreman via hammer or
b) A host with that name/IP already existed in Foreman, was deleted and when trying to create the new host, you got a conflict with old data that was not properly cleaned up.

If both of these do not apply, I am pretty much out of ideas what would cause this error.

11

I agree, it looks like there is a stale entry somewhere but not knowing foreman that well not sure where to look - any ideas?

We are running an Infloblox instance but an nslookup again shows no entries.

12

If you are running infoblox, why are you using omshell? I’m not sure if that’s actually compatible, but you might want to take a look at the smart-proxy infoblox DHCP plugin, we have been using that for many years now and not encountered any problems with it.
For any stale data on the infoblox side, you will probably want to take a look at the infoblox side of things directly and search there for any leftover data. Depending on how the interaction between omshell and infoblox works, I could imagine there being data left which might not respond to nslookup. Without a detailed description of your setup and a look at the infoblox, that’s really hard to tell what’s going on.

13

Infoblox is not being used for foreman, foreman is using a local dhcp service. But we have an infoblox service which runs in parallel for mainstream IT.

14

Alright, so that should not be the source of the problem. The question still remains: Where does the conflicting data come from?

Any idea on this?