Some problems with uefi discovery, need some help

dgibson · August 28, 2018, 7:36am

Problem:
We use Intel NUCs, the latest models no longer support legacy bios, so I need to get UEFI boot working for discovery and provisioning. I am trying with pxegrub2_discovery template

I only have the below problem with UEFI boot, legacy pxe (PXELinux global default) discovery is working fine

The discovery image netboots ok, but when it comes to the point where it prompts for the network configuration, and I let it timeout to automatically configure via DHCP it fails to find a suitable interface and does not get a lease
I get the following error on the console log

fdi /usr/bin/discovery-register[2414] Registering host at (https://foreman)
fdi /usr/bin/discovery-register[2414] Could not retrieve fact=‘ipaddress’, resolution=‘anonymous’ : Could not execute ‘host fdi’ : command not found
fdi /usr/bin/discovery-register[2414] Could not send facts to server: getaddrinfo: Name or service not known

if I monitor using wireshark for DHCP/BOOTP I see no requests coming from the discovery image

however if I choose DHCP from the menu and just hit enter through all the options it gets an address fine

Expected outcome:
Discovery image requests dhcp address when UEFI booting
Foreman and Proxy versions:
Foreman 1.17.3
Discovery v3.4.4 image
Foreman and Proxy plugin versions:
DNS plugin 1.17.3
Discovery plugin 1.0.3

Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]

lzap · August 28, 2018, 1:37pm

Wait a minute, when you netboot FDI it does not ask anything, it just registers. FDI looks for “IPAPPEND” kernel command line added by PXELinux or Grub2 template and then it just goes on. Not sure what kind of workflow this is.

Quick google finds that this is a failure coming from facter when package named net-tools is not installed. But this package is present in discovery image for years, including the version 3.4.4. Weird, were you tinkering with the FDI yourself (own build)? The other line looks like when facter is resolving ipaddress fact, it tries to resolve fdi hostname as well which lead to error. In that case I’d say your DHCP configuration is wrong.

Oh this just looks you messed up networking options, you need to provide a valid DNS server, looks like there is a “host fdi” call from facter which actually tries to resolve fdi and that fails for some reason.

I’d be interested however in seing output of “host fdi” after you get into this “weird” state. Can you enable ssh/root access via fdi.ssh=1 fdi.rootpw=somepass and run “facter ipaddress” and then “host fdi” manually?

dgibson · August 29, 2018, 11:16am

aaah, right. My boot menu doesn’t have IPAPPEND, I am assuming as it is efiboot it is using some different kernel parameters, including BOOTIF which should be set to the mac address.

menuentry 'Foreman Discovery Image' {
  linuxefi boot/fdi-image/vmlinuz0 rootflags=loop root=live:/fdi.iso rootfstype=auto ro rd.live.image acpi=force rd.luks=0 rd.md=0 rd.dm=0 rd.lvm=0 rd.bootif=0 rd.neednet=0 nomodeset proxy.url=https://foremantest.fritz.box proxy.type=foreman BOOTIF=01-$mac fdi.ssh=1 fdi.rootpw=pass
  initrdefi boot/fdi-image/initrd0.img
}

It looks like that isnt happening though, if I look at the value of the BOOTIF parameter on the booted failed discovery it shows it as 01-------
so looks like the regexp is not setting the value of the mac.
I changed the flag “rd.neednet=0” to 1 to force the network up. It now gets an ip and I now get the error

Response from server 422: {“message”:“ERF42-6513 [Foreman::Exception]: Unable to detect primary interface using MAC ‘:::::’ specified by discovery_fact ‘discovery_bootif’”}

Which seems to confirm that.
The grub.cfg in /var/lib/tftpboot/grub2 states
# On Debian/Ubuntu grub2 does not have patch for loading MAC-based configs. Also due to bug
# in RHEL 7.4 files are loaded with an extra “:” character at the end. This workarounds both
# cases, make sure “regexp.mod” file is present on the TFTP. For more info see:

I have it in /usr/lib/grub/x86_64-efi/regexp.mod does it need to be in /var/lib/tftpboot/grub2 too?
If so then are there other pieces missing ?

my grub2 folder has just grub.cfg, grubx64.efi and shim.efi

(Ubuntu 16.04 server btw)

thanks for the help

lzap · August 30, 2018, 11:41am

Deciding on that is I have to admit bad, we should do this more explicitly.

Do you have all the bits in place. You copied that correctly from:

github.com

theforeman/community-templates/blob/69923c46b1ca6c09c0677a781c7c2e3b90f7e522/provisioning_templates/snippet/pxegrub2_discovery.erb

<%#
kind: snippet
name: pxegrub2_discovery
model: ProvisioningTemplate
snippet: true
-%>
<%#
RHEL 7 virtio driver does not have persistent naming scheme, causing interfaces
to be named eth0, eth1, etc.. If you want to enable the new persistent naming
scheme and get inteface names like ens3, add net.ifnames=1 to the linuxefi line
below. This will not be fixed until RHEL8 due to prevent breaking changes for
existing systems. See RHBZ#1259015 for more details.

Another bug in RHEL 7.4 is affecting $net_default_mac with a trailing slash, in
the global template we have a regexp module creating corrected variable $mac
which is used here. See RHBZ#1487107 for more info.
-%>
menuentry 'Foreman Discovery Image' --id discovery {
  linuxefi boot/fdi-image/vmlinuz0 rootflags=loop root=live:/fdi.iso rootfstype=auto ro rd.live.image acpi=force rd.luks=0 rd.md=0 rd.dm=0 rd.lvm=0 rd.bootif=0 rd.neednet=0 nomodeset proxy.url=<%= foreman_server_url %> proxy.type=foreman BOOTIF=01-$mac
  initrdefi boot/fdi-image/initrd0.img

This file has been truncated. show original

But this is a snipped used in this context:

github.com

theforeman/community-templates/blob/69923c46b1ca6c09c0677a781c7c2e3b90f7e522/provisioning_templates/PXEGrub2/pxegrub2_global_default.erb#L11-L19


      
          # On Debian/Ubuntu grub2 does not have patch for loading MAC-based configs. Also due to bug
          # in RHEL 7.4 files are loaded with an extra ":" character at the end. This workarounds both
          # cases, make sure "regexp.mod" file is present on the TFTP. For more info see:
          # https://bugzilla.redhat.com/show_bug.cgi?id=1370642#c70
          insmod regexp
          regexp --set=1:m1 --set=2:m2 --set=3:m3 --set=4:m4 --set=5:m5 --set=6:m6 '^([0-9a-f]{1,2})\:([0-9a-f]{1,2})\:([0-9a-f]{1,2})\:([0-9a-f]{1,2})\:([0-9a-f]{1,2})\:([0-9a-f]{1,2})' "$net_default_mac"
          mac=${m1}-${m2}-${m3}-${m4}-${m5}-${m6}
          configfile=/grub2/grub.cfg-01-$mac
          source "$configfile"

See it’s actually setting the mac variable there to workaround a bug which came with RHEL 7.4 in grub2. Now its fixed so if you are on grub2 from 7.5+ you are good and you can simply use just $net_default_mac.

Cheers.

dgibson · August 30, 2018, 3:35pm

Thanks a mill for the pointers. my grub2 was looking for regexp.mod in /EFI/BOOT/x86_64-efi/regexp.mod
and not finding it.

Aug 30 17:21:13 foremantest in.tftpd[13860]: RRQ from 192.168.178.172 filename /grub2/grub.cfg
Aug 30 17:21:13 foremantest in.tftpd[13861]: RRQ from 192.168.178.172 filename /EFI/BOOT/x86_64-efi/regexp.mod
Aug 30 17:21:13 foremantest in.tftpd[13861]: sending NAK (1, File not found) to 192.168.178.172
Aug 30 17:21:13 foremantest in.tftpd[13862]: RRQ from 192.168.178.172 filename /grub2/grub.cfg-01------

Once I created that folder structure and copied the file there the regex worked

Aug 30 17:23:38 foremantest in.tftpd[13984]: RRQ from 192.168.178.172 filename /grub2/grub.cfg
Aug 30 17:23:38 foremantest in.tftpd[13985]: RRQ from 192.168.178.172 filename /EFI/BOOT/x86_64-efi/regexp.mod
Aug 30 17:23:38 foremantest in.tftpd[13986]: RRQ from 192.168.178.172 filename /grub2/grub.cfg-01-f4-4d-30-60-d1-fd

and fdi is now successfully registering the discovery in foreman

Bit of a hack, but it’s working. Now to figure out why my grub wants that file/folder structure…

thanks again!

lzap · September 3, 2018, 8:59am

Great, our installer is supposed to set this up for you. Were you using the installer?

dgibson · September 4, 2018, 8:33am

Yes, I used the installer. It didnt install anything into the grub2 folder other than the aforementioned grub.cfg, grubx64.efi and shim.efi(symlink to grubx64.efi)

I’ve messed about a bit with the config and upgraded to 1.18 in the meantime, so I can’t say for sure what a clean 1.18 looks like but I’m going to do a fresh install and I’ll report back

lzap · September 4, 2018, 4:54pm

Can you please make a symlink that will make it to use regexp.mod on the correct path so it works also for Ubuntu Server out of box? Then we can add this to our installer so it works out of box.

I’d assume EFI/BOOT/x86_64-efi → grub2/ but can you confirm?

A change needs to be done in GitHub - theforeman/puppet-foreman_proxy: Puppet module for Foreman Smart Proxy if you want to contribute.

ekohl · November 19, 2018, 2:00pm

Didn’t this solve it?

kariste · August 28, 2019, 7:56am

I faced the same problem. The solution was the following:
mkdir -p /srv/tftp/EFI/BOOT/x86_64-efi
cp /usr/lib/grub/x86_64-efi/regexp.mod /srv/tftp/EFI/BOOT/x86_64-efi

@ekohl No, this fix doesn’t address the same problem.

lzap · August 29, 2019, 12:54pm

Foreman 1.24 will now also deploy Grub2 configuration in the form of grub2.cfg-00:11:22:33:44:55:66 so no regular expression tricks will need to happen anymore. The template was updated to simply use MAC address grub2 predefined variable.