Foreman 1.20 / Host creation fails to provision dhcp entry

Problem:
When creating a multi-homed host, foreman creates one the DHCP entry for the first interface.

Expected outcome:
Dhcp entries are created for both entries.

Foreman and Proxy versions:
1.20.1 both

Foreman and Proxy plugin versions:

Other relevant data:
dhcp entries are created in the infoblox using
rubygem-smart_proxy_dhcp_infoblox-0.0.13-1.fm1_18.el7.noarch
rubygem-infoblox-2.0.4-1.el7.noarch
In logs everything looks ok.
Our setup was working ok using foreman 1.17. We had to upgrade to 1.20 to keep using a supported version.

Is there any known regression or featureā€™s change that could explain to this behaviour ?
How to restore the previous behaviour ?

Thanks for helping.

I have just checken on our 1.20 instance and it works fine. The same infoblox related packages are installed on our system as well.

Did you change anything else? Maybe forgot to set managed for the second interface? Have you created a new network and not set the apropriate DHCP proxy for that?
Just poking around with the things that pop info my mind.

Thanks for the reply.
This setup used to worked on Foreman 1.17.
If we create a physical host with mac addresses of each interface, both interface gets provisionned through the DHCP.
It blocks only when creating hosts through fog/vmware.
In this case, the mac adresses are not known. They are automatically assigned by the vcenter and sent back to foreman which completes the host informations with the MAC addresses.
If one modifies the second interface, and update the hosts, then the DHCP entry for the second interface gets created.

So you have a host with two managed interface and two names (in different or same domain). Only one NIC gets the DHCP record created.

Thatā€™s correct.
We use the infoblox DHCP plugin to create host record in our DNS,
Only 1 record gets created.

I just doublechecked here.
Environment:

  • Foreman 1.20.1
  • Foreman Smart Proxy 1.20.1
  • infoblox related packages: see above
  • VCenter 6.7

DHCP Smartproxy for Infoblox creates Host records in the infoblox with DNS and DHCP records. I create a multihomed VM in VMware, I can see both records in the infoblox UI directly afterwards, containing IP, DNS and MAC.

Okay the problem seems to be elsewhere.
What could cause the dhcp creation to be skipped ?

The only things I know of that cause record creation to be skipped is if either
a) the managed flag is not set on the corresponding interface or
b) the subnet is not associated with your DHCP smartproxys.

To my knowledge and experience, everything als should result in some kind of error in UI and/or logs (production.log or proxy.log).

There are couple of checks :slight_smile:

  def dhcp?
    # host.managed? and managed? should always come first so that orchestration doesn't
    # even get tested for such objects
    #
    # The subnet boot mode is ignored as DHCP can be required for PXE or image provisioning
    # steps, while boot mode can be used in templates later.
    (host.nil? || host.managed?) && managed? && hostname.present? && ip_available? && mac_available? &&
        !subnet.nil? && subnet.dhcp? && SETTINGS[:unattended] && (!provision? || operatingsystem.present?)
  end

Iā€™m sharing an extract of the logs from a host creation I did a few hours ago.
I redacted all the private information.
https://drive.google.com/open?id=1DTjMNVE9lzkRs69WbCzejOt3BwytQsSO
I put a debug line before the checks to reports in which state are the checks.
Iā€™m afraid that nothing seems wrong.
I think we need to search the root cause in the callers stack.
We worked around the problem by using the DNS infoblox plugin, but as you can see, the creation is painfully slow.

Not sure what happened here, but after trying to reproduce again today (did not change a thing in the meantime) I am suddenly able to reproduce this. I can not tell for sure whether I was dreaming last time I tried it (what is what I guess) or it really just stopped working, but with VMWare compute I can now confirm this behaviour.

Looks like the ā€œCreate DHCP Settingā€ orchestration event does not get called/created for additional VMWare interfaces. Iā€™m not familiar enough with the code and Rail/ActiveRecord to even take a guess where this is going wrong. Why it only happens on VMWare and not with baremetal confuses me too.

Can you do this once again and enable debug mode of Rails in settings.yaml? Because orchestration puts many debug messages there, the info message indicates that there were 6 steps scheduled (Processed 6 tasks from queue 'Host::Managed Main', completed 6/6) but there is not enough details.

There is also this line Create DHCP reservation jana-barcelo... which indicates there was an attempt to call DHCP module on proxy. Can you confirm in smart-proxy.log there was a request?

The problem is, there should be two such entries. When I create a physical machine, Foreman calls that twice, once for each interface. I can then see two ā€œCreate DHCP reservationā€¦ā€ messages in the logs.

So, I enabled foreman debugging and fired up a new VM.
Since just about all pastebin services I could find are blocked at our corporate network (thanks stupid blacklisting systems) I just put it up as a gitlab snippet: https://gitlab.com/snippets/1826253
I hope I enabled all the necessary debug options. If not, please tell me whats missing.

Now I see the problem:

2019-02-19T15:56:04 [D|app|bd1f5] Task 'dhcp_create_' already in 'Host::Managed Main' queue

I am the guilty one. There was a patch in Foreman which prevents from creating multiple DHCP create orchestration steps because we had bugs when this was scheduled multiple times: Bug #21120: Multiple DHCP orchestration is no longer possible with PXELoader - Foreman

The unique identifier is created as: dhcp_create_#{self.mac} and obviously in your case thatā€™s nil - empty string. Therefore both NICs have the same orchestration identifier and the second attempt is ignored.

Itā€™s getting late here, I can fix that later. Can you share what exactly do you have set for your NICs? We cannot use MAC address obviously, can we use identifier (e.g. eth0)? If not that database ID could be a fallback as well, but it can also be nil sometimes I guess - when record hasnā€™t been saved yet?

Dirty patch:

Replace dhcp_*_#{self.mac} with dhcp_*_#{self.mac | self.identifier | self.id} in dhcp.rb:

https://github.com/theforeman/foreman/pull/5079/files

Hi,

thanks for the reply.
We actually do not have identifier set either. Usually, at our site we set the bare minimum of options required and rely on orchestration and puppet to report missing information.
We only set:

  • Type
  • DNS name
  • IPv4 Subnet
  • IPv4 IP (usually recieved directly from smartproxy)
  • The Primary/Managed/etc. checkboxes as required

The only thing I can see here that should always be unique in my opinion would be IPv4 adress.

I will take a look at the patch later today when Iā€™m done with all the meetings I have to attend today.

You proposal doesnā€™t work. It fails wiith the error message:
NoMethodError: undefined method `|ā€™ for ā€œ00:50:56:8e:be:6eā€:String
I made a patch using your directions but using a private function instead.
I tested it in testing env, it creates one host and deletes it ok.
Hereā€™s a pull request of the modfication Pull request #6497

Great, please file a PR against develop branch first, make sure to create a RedMine ticket and create the commit accordingly: Foreman :: Contribute

I suggest to rename the method select_attribute to generate_id(action) and return whole string in the form of "dhcp_#{action}_#{unique_id}". I think we should create this fallback chain:

unique_id = self.mac | self.ip | self.identifier | self.id

That should cover all use cases when something is missing.

Hum, Iā€™m not a ruby expert, here, but if mac address exists and is a string, you end up with the same error I mentioned in previous post, that ā€˜|ā€™ method does not exists for the string object which is indeed true.
This is the reason why I created a function.
Iā€™ll figure out something that work if Iā€™m able to do it.

Easiest and most compact would be probably (havent tested):

unique_id = [self.mac, self.ip, self.identifier, self.id].find{|x| x&.present?}