Provisioning bare metal host over UEFI. Client requesting grub.cfg in wrong location

jbrown8380 · June 16, 2022, 10:52pm

Problem:
I’m trying to provision and install Rocky 8.5 onto a bare metal host. This host’s BIOS does not provide a “Legacy BIOS” option for booting. Therefore, I am stuck supporting a UEFI solution. Upon powering on the bare metal host, I can see that it properly obtains it’s IP address and next-server over DHCP. It also successfully pulls the grub2/grubx64.efi file over TFTP.

It then requests the “/EFI/rocky/grub.cfg-MACDDRESS” file over TFTP from the foreman-proxy. However, that file is not found in that location under the tftpboot root directory on the foreman-proxy.

Foreman creates the “grub.cfg-MACADDRESS” file and places it in the “/grub2” directory under the tftp root.

How do I get the host to request its “grub.cfg-MACADDRESS” file from the /grub2 directory?

Expected outcome:
After the host has pulled the grub2/grubx64.efi from the foreman-proxy, it should then pull the “grub.cfg-MACADDRESS” file from the grub2 directory as well. It should not be looking in a non-existent “/EFI/rocky/” directory.

Foreman and Proxy versions:
Both Foreman and Foreman-Proxy are on 3.1.2

Foreman and Proxy plugin versions:
No plugins are installed.
The Foreman-Proxy has the tftp, dhcp, Logs, and Templates features enabled.

Distribution and version:
Rocky Linux version 8.5

Other relevant data:

Packet capture on the Foreman-Proxy at the time in which the host is powered on:


IP (tos 0x0, ttl 64, id 41497, offset 0, flags [none], proto UDP (17), length 375)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from a4:bb:6d:81:73:3f, length 347, xid 0xe102f083, secs 28, Flags [Broadcast] (0x8000)
	  Client-Ethernet-Address a4:bb:6d:81:73:3f
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    MSZ Option 57, length 2: 1472
	    Parameter-Request Option 55, length 35: 
	      Subnet-Mask, Time-Zone, Default-Gateway, Time-Server
	      IEN-Name-Server, Domain-Name-Server, Hostname, BS
	      Domain-Name, RP, EP, RSZ
	      TTL, BR, YD, YS
	      NTP, Vendor-Option, Requested-IP, Lease-Time
	      Server-ID, RN, RB, Vendor-Class
	      TFTP, BF, GUID, Option 128
	      Option 129, Option 130, Option 131, Option 132
	      Option 133, Option 134, Option 135
	    GUID Option 97, length 17: 0.68.69.76.76.76.0.16.66.128.50.195.192.79.75.68.51
	    NDI Option 94, length 3: 1.3.16
	    ARCH Option 93, length 2: 7
	    Vendor-Class Option 60, length 32: "PXEClient:Arch:00007:UNDI:003016"
	    END Option 255, length 0
IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 337)
    100.99.97.7.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 309, xid 0xe102f083, secs 28, Flags [Broadcast] (0x8000)
	  Your-IP 100.99.97.48
	  Server-IP 100.99.97.7
	  Client-Ethernet-Address a4:bb:6d:81:73:3f
	  file "grub2/grubx64.efi"
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Offer
	    Server-ID Option 54, length 4: 100.99.97.7
	    Lease-Time Option 51, length 4: 43200
	    Subnet-Mask Option 1, length 4: 255.255.0.0
	    Default-Gateway Option 3, length 4: 100.99.0.1
	    Domain-Name-Server Option 6, length 8: 100.99.97.4,100.99.97.5
	    Hostname Option 12, length 20: "workstation1.mts.sys"
	    Domain-Name Option 15, length 7: "mts.sys"
	    END Option 255, length 0
IP (tos 0x0, ttl 64, id 41498, offset 0, flags [none], proto UDP (17), length 387)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from a4:bb:6d:81:73:3f, length 359, xid 0xe102f083, secs 28, Flags [Broadcast] (0x8000)
	  Client-Ethernet-Address a4:bb:6d:81:73:3f
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Request
	    Server-ID Option 54, length 4: 100.99.97.7
	    Requested-IP Option 50, length 4: 100.99.97.48
	    MSZ Option 57, length 2: 65280
	    Parameter-Request Option 55, length 35: 
	      Subnet-Mask, Time-Zone, Default-Gateway, Time-Server
	      IEN-Name-Server, Domain-Name-Server, Hostname, BS
	      Domain-Name, RP, EP, RSZ
	      TTL, BR, YD, YS
	      NTP, Vendor-Option, Requested-IP, Lease-Time
	      Server-ID, RN, RB, Vendor-Class
	      TFTP, BF, GUID, Option 128
	      Option 129, Option 130, Option 131, Option 132
	      Option 133, Option 134, Option 135
	    GUID Option 97, length 17: 0.68.69.76.76.76.0.16.66.128.50.195.192.79.75.68.51
	    NDI Option 94, length 3: 1.3.16
	    ARCH Option 93, length 2: 7
	    Vendor-Class Option 60, length 32: "PXEClient:Arch:00007:UNDI:003016"
	    END Option 255, length 0
IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 337)
    100.99.97.7.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 309, xid 0xe102f083, secs 28, Flags [Broadcast] (0x8000)
	  Your-IP 100.99.97.48
	  Server-IP 100.99.97.7
	  Client-Ethernet-Address a4:bb:6d:81:73:3f
	  file "grub2/grubx64.efi"
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: ACK
	    Server-ID Option 54, length 4: 100.99.97.7
	    Lease-Time Option 51, length 4: 43200
	    Subnet-Mask Option 1, length 4: 255.255.0.0
	    Default-Gateway Option 3, length 4: 100.99.0.1
	    Domain-Name-Server Option 6, length 8: 100.99.97.4,100.99.97.5
	    Hostname Option 12, length 20: "workstation1.mts.sys"
	    Domain-Name Option 15, length 7: "mts.sys"
	    END Option 255, length 0
IP (tos 0x0, ttl 64, id 41499, offset 0, flags [none], proto UDP (17), length 88)
    100.99.97.48.1822 > 100.99.97.7.69: [udp sum ok]  60 RRQ "grub2/grubx64.efi" octet tsize 0 blksize 1468 windowsize 4
IP (tos 0x0, ttl 64, id 41501, offset 0, flags [none], proto UDP (17), length 80)
    100.99.97.48.1823 > 100.99.97.7.69: [udp sum ok]  52 RRQ "grub2/grubx64.efi" octet blksize 1468 windowsize 4
IP (tos 0x0, ttl 255, id 9217, offset 0, flags [none], proto UDP (17), length 98)
    100.99.97.48.25300 > 100.99.97.7.69: [udp sum ok]  70 RRQ "/EFI/rocky/grub.cfg-01-a4-bb-6d-81-73-3f" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9218, offset 0, flags [none], proto UDP (17), length 86)
    100.99.97.48.25301 > 100.99.97.7.69: [udp sum ok]  58 RRQ "/EFI/rocky/grub.cfg-64636130" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9219, offset 0, flags [none], proto UDP (17), length 85)
    100.99.97.48.25302 > 100.99.97.7.69: [udp sum ok]  57 RRQ "/EFI/rocky/grub.cfg-6463613" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9220, offset 0, flags [none], proto UDP (17), length 84)
    100.99.97.48.25303 > 100.99.97.7.69: [udp sum ok]  56 RRQ "/EFI/rocky/grub.cfg-646361" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9221, offset 0, flags [none], proto UDP (17), length 83)
    100.99.97.48.25304 > 100.99.97.7.69: [udp sum ok]  55 RRQ "/EFI/rocky/grub.cfg-64636" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9222, offset 0, flags [none], proto UDP (17), length 82)
    100.99.97.48.25305 > 100.99.97.7.69: [udp sum ok]  54 RRQ "/EFI/rocky/grub.cfg-6463" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9223, offset 0, flags [none], proto UDP (17), length 81)
    100.99.97.48.25306 > 100.99.97.7.69: [udp sum ok]  53 RRQ "/EFI/rocky/grub.cfg-646" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9224, offset 0, flags [none], proto UDP (17), length 80)
    100.99.97.48.25307 > 100.99.97.7.69: [udp sum ok]  52 RRQ "/EFI/rocky/grub.cfg-64" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9225, offset 0, flags [none], proto UDP (17), length 79)
    100.99.97.48.25308 > 100.99.97.7.69: [udp sum ok]  51 RRQ "/EFI/rocky/grub.cfg-6" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9226, offset 0, flags [none], proto UDP (17), length 77)
    100.99.97.48.25309 > 100.99.97.7.69: [udp sum ok]  49 RRQ "/EFI/rocky/grub.cfg" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9227, offset 0, flags [none], proto UDP (17), length 91)
    100.99.97.48.25310 > 100.99.97.7.69: [udp sum ok]  63 RRQ "/EFI/rocky/x86_64-efi/command.lst" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9228, offset 0, flags [none], proto UDP (17), length 86)
    100.99.97.48.25311 > 100.99.97.7.69: [udp sum ok]  58 RRQ "/EFI/rocky/x86_64-efi/fs.lst" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9229, offset 0, flags [none], proto UDP (17), length 90)
    100.99.97.48.25312 > 100.99.97.7.69: [udp sum ok]  62 RRQ "/EFI/rocky/x86_64-efi/crypto.lst" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9230, offset 0, flags [none], proto UDP (17), length 92)
    100.99.97.48.25313 > 100.99.97.7.69: [udp sum ok]  64 RRQ "/EFI/rocky/x86_64-efi/terminal.lst" octet blksize 1024 tsize 0

Directory structure of /var/lib/tftpboot on the Foreman-Proxy (other, already working, host configuration files excluded):
.
├── boot
│ ├── rocky8-5-MxYLsytBkvSG-initrd.img
│ ├── rocky8-5-MxYLsytBkvSG-vmlinuz
├── chain.c32
├── grub
├── grub2
│ ├── boot → /var/lib/tftpboot/boot
│ ├── grub.cfg
│ ├── grub.cfg-01-a4-bb-6d-81-73-3f
│ ├── grub.cfg-a4:bb:6d:81:73:3f
│ ├── grubx64.efi
│ ├── shim.efi → /var/lib/tftpboot/grub2/shimx64.efi
│ └── shimx64.efi
├── ldlinux.c32
├── libcom32.c32
├── libutil.c32
├── memdisk
├── menu.c32
├── poap.cfg
├── pxelinux.0
├── pxelinux.cfg
│ └── 01-a4-bb-6d-81-73-3f
└── ztp.cfg

jbrown8380 · June 16, 2022, 10:58pm

Here’s what my PXEGrub2 template looks like:

# This file was deployed via 'Kickstart default PXEGrub2 clone' template

set default=0
set timeout=10

menuentry 'Kickstart default PXEGrub2 clone' {
  linuxefi boot/rocky8-5-MxYLsytBkvSG-vmlinuz  BOOTIF=01-a4-bb-6d-81-73-3f ks=http://inf-4.mts.sys:8000/unattended/provision?token=6cabdf16-2005-4c21-9282-d2b362b8a5d4 kssendmac ks.sendmac ip=dhcp nameserver=100.99.97.4 nameserver=100.99.97.5
  initrdefi boot/rocky8-5-MxYLsytBkvSG-initrd.img
}

lzap · June 21, 2022, 6:56am

It smells like Rocky people are building the grub with incorrect options, it should respect the fact that it was booted over network and grab the config from the relative path.

It always worked fine on RHEL, maybe there was some regression in 8.5 I haven’t heard about tho. Can you take the grubx64.efi from CentOS 8 Stream unpacking the RPM and trying again to rule out a build issue?

Or better use the build from Fedora Rawhide.

jbrown8380 · June 21, 2022, 2:44pm

Thanks for your response @lzap,

I have indeed learned recently that the grubx64.efi bootloader is compiled with an option that defines the “prefix” of where it should look to grab it’s grub.cfg file. So I agree with your suspicion that the Rocky grubx64.efi might be built differently from the CentOS 8 Stream version (or some other distribution) of the file.

Nevertheless, I’ve managed to get it to boot by simply adding an “EFI” directory under /var/lib/tftpboot and then adding a “rocky” symbolic link pointing to “…/grub2”. Even though it’s evident in the packet capture that it’s requesting what appears to be an absolute path (with the preceding / in front of EFI), it still seems to be relative to the tftpboot root dir. So with this tweak, I can get by.

However, I’ll still give your suggestion a try and grab the grubx64.efi file from the CentOS 8 Stream RPM. I’ll report back with the results.

jbrown8380 · June 21, 2022, 4:34pm

@lzap,

I’ve extracted the grubx64.efi file from the grub2-efi-x64-2.02-123.el8.x86_64.rpm provided in the CentOS 8 Stream repos and placed it in the /var/lib/tftpboot/grub2 directory on my Foreman server. I then instructed Foreman to rebuild my workstation on the next boot. From there I performed another packet capture and this time I saw the client attempting to pull it’s grub.cfg-MACADDRESS file from the /EFI/centos directory. So, effectively, the same problem was present. Of couse, I could use my tweak from above and create a symbolic link that would allow the client to download its file in the grub2 directory instead.

Here’s a copy of the packet capture again:

IP (tos 0x0, ttl 64, id 32797, offset 0, flags [none], proto UDP (17), length 375)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from a4:bb:6d:81:73:3f, length 347, xid 0x2143ce88, secs 12, Flags [Broadcast] (0x8000)
	  Client-Ethernet-Address a4:bb:6d:81:73:3f
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Discover
	    MSZ Option 57, length 2: 1472
	    Parameter-Request Option 55, length 35: 
	      Subnet-Mask, Time-Zone, Default-Gateway, Time-Server
	      IEN-Name-Server, Domain-Name-Server, Hostname, BS
	      Domain-Name, RP, EP, RSZ
	      TTL, BR, YD, YS
	      NTP, Vendor-Option, Requested-IP, Lease-Time
	      Server-ID, RN, RB, Vendor-Class
	      TFTP, BF, GUID, Option 128
	      Option 129, Option 130, Option 131, Option 132
	      Option 133, Option 134, Option 135
	    GUID Option 97, length 17: 0.68.69.76.76.76.0.16.66.128.50.195.192.79.75.68.51
	    NDI Option 94, length 3: 1.3.16
	    ARCH Option 93, length 2: 7
	    Vendor-Class Option 60, length 32: "PXEClient:Arch:00007:UNDI:003016"
	    END Option 255, length 0
IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 337)
    100.99.97.7.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 309, xid 0x2143ce88, secs 12, Flags [Broadcast] (0x8000)
	  Your-IP 100.99.97.38
	  Server-IP 100.99.97.7
	  Client-Ethernet-Address a4:bb:6d:81:73:3f
	  file "grub2/grubx64.efi"
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Offer
	    Server-ID Option 54, length 4: 100.99.97.7
	    Lease-Time Option 51, length 4: 43200
	    Subnet-Mask Option 1, length 4: 255.255.0.0
	    Default-Gateway Option 3, length 4: 100.99.0.1
	    Domain-Name-Server Option 6, length 8: 100.99.97.4,100.99.97.5
	    Hostname Option 12, length 20: "workstation1.mts.sys"
	    Domain-Name Option 15, length 7: "mts.sys"
	    END Option 255, length 0
IP (tos 0x0, ttl 64, id 32798, offset 0, flags [none], proto UDP (17), length 387)
    0.0.0.0.68 > 255.255.255.255.67: [udp sum ok] BOOTP/DHCP, Request from a4:bb:6d:81:73:3f, length 359, xid 0x2143ce88, secs 12, Flags [Broadcast] (0x8000)
	  Client-Ethernet-Address a4:bb:6d:81:73:3f
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: Request
	    Server-ID Option 54, length 4: 100.99.97.7
	    Requested-IP Option 50, length 4: 100.99.97.38
	    MSZ Option 57, length 2: 65280
	    Parameter-Request Option 55, length 35: 
	      Subnet-Mask, Time-Zone, Default-Gateway, Time-Server
	      IEN-Name-Server, Domain-Name-Server, Hostname, BS
	      Domain-Name, RP, EP, RSZ
	      TTL, BR, YD, YS
	      NTP, Vendor-Option, Requested-IP, Lease-Time
	      Server-ID, RN, RB, Vendor-Class
	      TFTP, BF, GUID, Option 128
	      Option 129, Option 130, Option 131, Option 132
	      Option 133, Option 134, Option 135
	    GUID Option 97, length 17: 0.68.69.76.76.76.0.16.66.128.50.195.192.79.75.68.51
	    NDI Option 94, length 3: 1.3.16
	    ARCH Option 93, length 2: 7
	    Vendor-Class Option 60, length 32: "PXEClient:Arch:00007:UNDI:003016"
	    END Option 255, length 0
IP (tos 0x10, ttl 128, id 0, offset 0, flags [none], proto UDP (17), length 337)
    100.99.97.7.67 > 255.255.255.255.68: [udp sum ok] BOOTP/DHCP, Reply, length 309, xid 0x2143ce88, secs 12, Flags [Broadcast] (0x8000)
	  Your-IP 100.99.97.38
	  Server-IP 100.99.97.7
	  Client-Ethernet-Address a4:bb:6d:81:73:3f
	  file "grub2/grubx64.efi"
	  Vendor-rfc1048 Extensions
	    Magic Cookie 0x63825363
	    DHCP-Message Option 53, length 1: ACK
	    Server-ID Option 54, length 4: 100.99.97.7
	    Lease-Time Option 51, length 4: 43200
	    Subnet-Mask Option 1, length 4: 255.255.0.0
	    Default-Gateway Option 3, length 4: 100.99.0.1
	    Domain-Name-Server Option 6, length 8: 100.99.97.4,100.99.97.5
	    Hostname Option 12, length 20: "workstation1.mts.sys"
	    Domain-Name Option 15, length 7: "mts.sys"
	    END Option 255, length 0
IP (tos 0x0, ttl 64, id 32799, offset 0, flags [none], proto UDP (17), length 88)
    100.99.97.38.1557 > 100.99.97.7.69: [udp sum ok]  60 RRQ "grub2/grubx64.efi" octet tsize 0 blksize 1468 windowsize 4
IP (tos 0x0, ttl 64, id 32801, offset 0, flags [none], proto UDP (17), length 80)
    100.99.97.38.1558 > 100.99.97.7.69: [udp sum ok]  52 RRQ "grub2/grubx64.efi" octet blksize 1468 windowsize 4
IP (tos 0x0, ttl 255, id 9217, offset 0, flags [none], proto UDP (17), length 99)
    100.99.97.38.25300 > 100.99.97.7.69: [udp sum ok]  71 RRQ "/EFI/centos/grub.cfg-01-a4-bb-6d-81-73-3f" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9218, offset 0, flags [none], proto UDP (17), length 87)
    100.99.97.38.25301 > 100.99.97.7.69: [udp sum ok]  59 RRQ "/EFI/centos/grub.cfg-64636126" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9219, offset 0, flags [none], proto UDP (17), length 86)
    100.99.97.38.25302 > 100.99.97.7.69: [udp sum ok]  58 RRQ "/EFI/centos/grub.cfg-6463612" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9220, offset 0, flags [none], proto UDP (17), length 85)
    100.99.97.38.25303 > 100.99.97.7.69: [udp sum ok]  57 RRQ "/EFI/centos/grub.cfg-646361" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9221, offset 0, flags [none], proto UDP (17), length 84)
    100.99.97.38.25304 > 100.99.97.7.69: [udp sum ok]  56 RRQ "/EFI/centos/grub.cfg-64636" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9222, offset 0, flags [none], proto UDP (17), length 83)
    100.99.97.38.25305 > 100.99.97.7.69: [udp sum ok]  55 RRQ "/EFI/centos/grub.cfg-6463" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9223, offset 0, flags [none], proto UDP (17), length 82)
    100.99.97.38.25306 > 100.99.97.7.69: [udp sum ok]  54 RRQ "/EFI/centos/grub.cfg-646" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9224, offset 0, flags [none], proto UDP (17), length 81)
    100.99.97.38.25307 > 100.99.97.7.69: [udp sum ok]  53 RRQ "/EFI/centos/grub.cfg-64" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9225, offset 0, flags [none], proto UDP (17), length 80)
    100.99.97.38.25308 > 100.99.97.7.69: [udp sum ok]  52 RRQ "/EFI/centos/grub.cfg-6" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9226, offset 0, flags [none], proto UDP (17), length 78)
    100.99.97.38.25309 > 100.99.97.7.69: [udp sum ok]  50 RRQ "/EFI/centos/grub.cfg" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9227, offset 0, flags [none], proto UDP (17), length 92)
    100.99.97.38.25310 > 100.99.97.7.69: [udp sum ok]  64 RRQ "/EFI/centos/x86_64-efi/command.lst" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9228, offset 0, flags [none], proto UDP (17), length 87)
    100.99.97.38.25311 > 100.99.97.7.69: [udp sum ok]  59 RRQ "/EFI/centos/x86_64-efi/fs.lst" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9229, offset 0, flags [none], proto UDP (17), length 91)
    100.99.97.38.25312 > 100.99.97.7.69: [udp sum ok]  63 RRQ "/EFI/centos/x86_64-efi/crypto.lst" octet blksize 1024 tsize 0
IP (tos 0x0, ttl 255, id 9230, offset 0, flags [none], proto UDP (17), length 93)
    100.99.97.38.25313 > 100.99.97.7.69: [udp sum ok]  65 RRQ "/EFI/centos/x86_64-efi/terminal.lst" octet blksize 1024 tsize 0

lzap · June 23, 2022, 10:41am

Thanks so this looks like a regression in RHEL/CentOS.

The thing is - in Foreman we use the official grub2 binary which comes from the grub2-efi package. It is meant for local booting and not network booting. So why we use it? It is simple - it is the only version which is signed by Red Hat (or CentOS/Fedora/any clone) therefore it works seamlessly with SecureBoot.

We could start building our grub via grub2-mknetdir which has options to configure where grub should load configuration files from, but then the bootloader will not be signed and SecureBoot will not be possible.

Therefore I suggest that we workaround the problem in the installer:

I emailed Red Hat grub2 maintainers about this to get more info on what happened, to me it looks like a security patch. Let’s see what they say, we will solve this.

WORKAROUND: The symlink as you figured out.

Thanks for the report and insights, this was REALLY hepful!

lzap · July 7, 2022, 1:12pm

Hey, we struggle to reproduce this locally, can you confirm that the crypto.lst was the last file before grub2 gets stuck?

We know that grub2 often requests various files on wrong paths and these are reported as “not found” by the TFTP server, however, they will not lead to immediate error. My concern is that we are focusing on an error which is actually not the root cause - might be something different (bug in PXE driver in firmware, corrupted kernel/initramdisk). What was the error you saw after the tcpdump capture?

jbrown8380 · July 7, 2022, 2:24pm

Hey @lzap,
The last file it requested wasn’t crypto.lst, but rather terminal.lst. I didn’t actually see an error at the end of the tcpdump capture, the client just dropped into the grub2 shell. However, it is possible I just didn’t see an error in time before it dropped into the grub2 shell.

I suppose it is possible that it could be the NIC firmware. The client with which I’m using to test is a fairly new Dell 7090 workstation. We’ve found that in order for these machines to support dual monitors, we have to upgrade the kernel to the kernel-ml in the elrepo-kernel repository. But that’s specifically a graphics card issue. We haven’t identified any kernel/firmware issues with the NICs. And upgrading the kernel would be a secondary step after we’ve provisioned the machine with Foreman. The kernel being loaded during provisioning with foreman would be the vanilla kernel that comes with the installation media.

ekohl · July 8, 2022, 12:52pm

I did find that I needed this patch to properly provision a Rocky Linux 8 machine:
https://github.com/theforeman/foreman/pull/9295

However, I didn’t need anything else when I tried to boot a VM with UEFI enabled. I should note that I had secure boot disabled, otherwise I couldn’t get it to boot.

Can you share a bit more about your environment?

For example, are you running Foreman on Rocky Linux too? Have you updated to Rocky Linux 8.6 by now?

Is Secure Boot enabled or disabled?

When does it fail? During startup of the installer or after the installer finished (that’s where it failed for me) when chainloading into the real OS.