Dhcp-isc auto-suggesting already assigned IPs - Foreman 3.0

Support
,
#1

Problem:
IP addresses already leased are being suggested during a provision.

Expected outcome:
IP addresses not already leased should be suggested

Foreman and Proxy versions:

Foreman and Proxy plugin versions:
3.0.1 both

Distribution and version:
Ubuntu 20.04

Other relevant data:

Foreman proxy says the IP is unused but if I manually check the lease file, it IS used.

2021-11-08T17:03:30 9bc2ff9c [I] Started GET /dhcp/192.168.250.0/unused_ip
2021-11-08T17:03:32 9bc2ff9c [I] Finished GET /dhcp/192.168.250.0/unused_ip with 200 (2005.5 ms)

Provisioning fails as expected as the DHCP cannot add the IP as it is already leased.

#2

Hello,

show me. When this request is processed, make a curl https://proxy:8443/dhcp/1.2.3.4 and make a copy of dhcpd.leases too.

Note Foreman supports “upgrading a lease to reservation”. If the host sends “preferred IP” address that belongs to the same MAC within the specified range, the lease is converted to reservation. Generally, we do not recommend the lease pool to overlap with Foreman subnet range tho.

You will find this tool useful when curing proxy because of SSL: Introducing fp-curl

#3

Here is the relevant information:

curl http://foreman.DN:8000/dhcp/192.168.250.153
{"reservations":[],"leases":[]}

And this was the already existing lease entry

lease 192.168.250.153 {
  starts 1 2021/10/11 18:24:37;
  ends 2 2021/10/12 06:24:37;
  tstp 2 2021/10/12 06:24:37;
  cltt 1 2021/10/11 18:24:37;
  binding state free;
  hardware ethernet 00:50:56:b2:42:5e;
}

It seems to me that the foreman proxy isn’t reading through the lease file.

#4

That is really strange, can you check the leases file is included in the dhcpd.conf? Our parser is really simple, but it understands “include” statement, maybe there is some weird syntax.

What is in your /etc/foreman-proxy/settings.d/dhcp_isc.yaml ? Can you doublecheck the leases path?

#5

What is your network mask? Note you must provide network address to the curl, just doublechecking. It is uncommon mask, that could be culprit of the problem.

#6 
sudo cat dhcp_isc.yml 
---
#
# Configuration file for ISC dhcp provider
#

:config: /etc/dhcp/dhcpd.conf
:leases: /var/lib/dhcp/dhcpd.leases

# Redhat 5
#
#:config: /etc/dhcpd.conf
#
# Settings for Ubuntu
#
#:config: /etc/dhcp3/dhcpd.conf
#:leases: /var/lib/dhcp3/dhcpd.leases

# Specifies TSIG key name and secret

#:key_name: secret_key_name
#:key_secret: secret_key


:omapi_port: 7911

# use :server setting in dhcp.yml if you are managing a dhcp server which is not localhost
#7 
cat dhcpd.conf 
# dhcpd.conf
omapi-port 7911;

default-lease-time 43200;
max-lease-time 86400;


not authoritative;


ddns-update-style none;

option domain-name "config.DOMAIN";
option domain-name-servers ALT IPS;
option ntp-servers none;

allow booting;
allow bootp;

option fqdn.no-client-update    on;  # set the "O" and "S" flag bits
option fqdn.rcode2            255;
option pxegrub code 150 = text ;





log-facility local7;

include "/etc/dhcp/dhcpd.hosts";
# config.DOMAIN
subnet 192.168.250.0 netmask 255.255.255.0 {
  pool
  {
    range 192.168.250.3 192.168.250.254;
  }

  option subnet-mask 255.255.255.0;
}
#8

Next question: what filesystem is your dhcpd.leases on? Does it support inotify? Do you happen to have a custom Linux kernel compiled without inotify support?

I suggest to perform a test via inotifywatch.

#9

The file system is ext4 using LVM

/dev/mapper/ubuntu--vg-ubuntu--lv ext4

The kernel is just a default Ubuntu Server Kernel from the official repo.

uname -a
Linux foreman-03 5.4.0-89-generic #100-Ubuntu SMP Fri Sep 24 14:50:10 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

inotifywatch doesn’t seem to do anything (even when I cat the file).

Was something changed in foreman 3.x from the 2.x regarding dhcpd leases? I never had a problem with getting IPs in the 2.x versions.

 sudo inotifywatch dhcpd.leases
Establishing watches...
Finished establishing watches, now collecting statistics.
#10

I see how inotifywatch does. I have to ctrl c it.

So after running inotifywatch and asking for a new IP I get this

sudo inotifywatch dhcpd.leases
Establishing watches...
Finished establishing watches, now collecting statistics.
^CNo events occurred.

But if a Run a grep on the file when using inotifywatch, than I get a result

sudo inotifywatch dhcpd.leases
Establishing watches...
Finished establishing watches, now collecting statistics.
^Ctotal  access  close_nowrite  open  filename
3      1       1              1     dhcpd.leases

So I think we can safely say, that the foreman-proxy is not accessing the lease file!!!

#11

And checking permissions

ll /var/lib/dhcp/dhcpd.leases
-rw-r--r-- 1 dhcpd dhcpd 60710 Nov 12 14:45 /var/lib/dhcp/dhcpd.leases

groups foreman-proxy
foreman-proxy : foreman-proxy dhcpd
#12

There were no changes I am aware of.

Anyways, we can find it. Can you turn on debug for smart proxy in /etc/foreman-proxy/settings.yaml and restart it? Then tail the proxy log, it should print something like:

caught :modify event on #{event.absolute_name}.

You should watch also for the Queue overflow occured when monitoring which would mean the inotify queue was full and messages were dropped.

#13

Here is the output from the debug.

2021-11-15T09:24:24 d15a4061 [W] Failed to add DHCP reservation for HOSTDN (192.168.250.166 / 00:50:56:84:d3:b9): No response from DHCP server: <Proxy::DHCP::Error>: Failed to add DHCP reservation for HOSTDN (192.168.250.166 / 00:50:56:84:d3:b9): No response from DHCP server
#14

Can you pastebin the whole transaction d15a4061? You can PM me if you have concerns about data.

#15

Thank you, I have emailed you.

#16

I did not get anything, please use lzap AT redhat dot com.

#17

Thank you, I have sent the email.

#18

Thanks, got it. For the record, if you use my github email remove the “-x” from the address, otherwise it goes to spam right next to those “job offers”. I recently created a new email lukasNOSPAM@zapletalovi.com which is hopefully more obvious.

The offending record is this one:

2021-11-22T09:14:46  [D] Added a reservation: 192.168.250.159:00:50:56:84:d1:d2:test-foreman-06.config.landcareresearch.co.nz

You can tell from the debug log:

omshell= name = "test-foreman-06.config.landcareresearch.co.nz"
omshell= ip-address = c0:a8:fa:d1
omshell= hardware-address = 00:50:56:84:78:19
omshell= hardware-type = 1
omshell= > obj: host
omshell= name = "test-foreman-06.config.landcareresearch.co.nz"
omshell= ip-address = c0:a8:fa:d1
omshell= hardware-address = 00:50:56:84:78:19
omshell= hardware-type = 1
omshell= statements = "filename = "pxelinux.0"; option host-name = "test-foreman-06.config.landcareresearch.co.nz";"
omshell= > can't open object: already exists
omshell= obj: host

Problem is not MAC or IP address, problem is the host name. See, in ISC DHCP, every reservation must have a name and foreman uses the FQDN for such name, ISC insists on the name to be unique as it is actually the identifier of the record.

You need to delete the offending record first which is a leftover from some kind of rename? or operation that did not succeed successfully.

#19

Thank you. These are for testing provisioning templates. I didn’t release they were in DHCP.

#20

For the record, we have script that should show you these offending records, it is all documented:

https://docs.theforeman.org/nightly/Provisioning_Guide/index-foreman-el.html#_troubleshooting_dhcp_problems_in_foreman

1 Like