Rediscovering hosts locally via Grub2

lzap · January 15, 2019, 8:07pm

When a host is managed via Foreman, it’s possible to download discovery image to it and create a Grub2 boot entry to quickly enter discovery mode again. This ad-hoc workflow requires the old managed host entry to be deleted prior provisioning discovered host, otherwise either “Name already taken” error can be shown or old and unused host entries will stack in the Foreman DB forever depending on the Foreman version in use.

This is a companion discussion topic for the original entry at https://theforeman.org/2019/01/rediscovering-hosts-via-grub2.html

Konstantin_Orekhov · January 16, 2019, 7:19pm

This has been a sticking point for us for a long time (and we have talked about this loophole in Foreman’s host management logic on numerous occasions). How this could be really automated? If managed machine crashes because it lost its disk, for example, and PXE-boots into FDI, I’d expect Foreman to properly deal with it as mark that system as rediscovered or failed or something to that degree, not just drop the ball with “Name already taken” message in a log file. If that new state was a proper state in Foreman, one could have execute some hook on that event to allow for more actions to be taken, possibly in automated fashion.

So, I guess, my question is when more than 2 states of the machine are expected to be in Foreman? Is there something already in the works?

Thanks!

lzap · January 17, 2019, 10:48am

We need to be very careful here since auto discovery rules could possibly initiate reprovisioning which could easily lead to data loss. That’s why I was rather staying away from this problem keeping the status quo.

However, we have improved this already a bit. There is a new Administer - Setting option “Error on existing NIC” and if you set it to true, a nicer error appear like “There is already existing managed host”. You can’t still easily rediscover. Feel free to tackle this problem:

github.com

theforeman/foreman_discovery/blob/develop/app/models/host/discovered.rb#L80-L99


      
          existing_managed = Nic::Managed.joins(:host).where(:mac => bootif_mac, :provision => true, :hosts => {:type => "Host::Managed"}).limit(1)
          if existing_managed.count > 0
            if Setting[:discovery_error_on_existing]
              raise ::Foreman::Exception.new("One or more existing managed hosts found: %s", "#{existing_managed.first.name}/#{bootif_mac}")
            else
              Rails.logger.warn("One or more existing managed hosts found: #{existing_managed.first.name}/#{bootif_mac}")
            end
          end
          
          # find existing discovered host (pick the oldest if multiple) or create new discovery host record
          existing_discovery_hosts = Nic::Managed.joins(:host).where(:mac => bootif_mac, :provision => true, :hosts => {:type => "Host::Discovered"}).order('created_at DESC')
          if existing_discovery_hosts.empty?
            host = Host.new(:name => hostname, :type => "Host::Discovered")
            send_notifications = true
          else
            Rails.logger.warn "Multiple (#{existing_discovery_hosts.count}) discovery hosts found with MAC address #{name_fact} - picking most recent NIC entry" if existing_discovery_hosts.count > 1
            host = existing_discovery_hosts.first.host
            send_notifications = false
          end

Please keep in mind that we need to disallow auto-provisioning when there was already an existing host by default to prevent data loss. Users should be allowed to opt-in tho via a setting or probably via some rule statement (we might add some fact for that).