PXE UEFI stopped working

Problem:
Client are not grabbing the custom grub MAC address client
Expected outcome:
Clients do download the custom grub MAC address client

Foreman and Proxy versions:
2.1.3

Distribution and version:
CentOS 7.8

Firstly, let me state this has been working for months. Two days ago, I noticed it is not.

We’re not doing many (if any) config changes in foreman. So I’m not sure if someone made a change, or something on the server side broke. But I did try to revert a backup from days earlier without success.

This is what’s happening: I boot a system and the console says:

Start PXE over IPv4
Station IP address is 192.168.10.71
Server IP address is 192.168.10.11
NBP filename is grub2/grubx64.efi
NBP filesize is 1097544 bytes
Downloading NBPfile..

NBP file downloaded successfully.  

Then, a very long wait, and then dumped into a grub> prompt.

The server logs show me this:

    Mar  9 08:49:02 s-ul11inf01 dhcpd: DHCPDISCOVER from 94:bb:6d:fb:ca:dd via em1
    Mar  9 08:49:02 s-ul11inf01 dhcpd: DHCPOFFER on 192.168.10.71 to 94:bb:6d:fb:ca:dd via em1
    Mar  9 08:49:05 s-ul11inf01 dhcpd: Dynamic and static leases present for 192.168.10.71.
    Mar  9 08:49:05 s-ul11inf01 dhcpd: from the dynamic address pool for 192.168.10.0/24
    Mar  9 08:49:05 s-ul11inf01 dhcpd: DHCPREQUEST for 192.168.10.71 (192.168.10.11) from 94:bb:6d:fb:ca:dd via em1
    Mar  9 08:49:05 s-ul11inf01 dhcpd: DHCPACK on 192.168.10.71 to 94:bb:6d:fb:ca:dd via em1
    Mar  9 08:49:05 s-ul11inf01 in.tftpd[215724]: RRQ from 192.168.10.71 filename grub2/grubx64.efi
    Mar  9 08:49:11 s-ul11inf01 in.tftpd[215724]: Client 192.168.10.71 timed out
    Mar  9 08:49:14 s-ul11inf01 in.tftpd[216173]: RRQ from 192.168.10.71 filename grub2/grubx64.efi
    Mar  9 08:49:14 s-ul11inf01 in.tftpd[216173]: Client 192.168.10.71 finished grub2/grubx64.efi

The grub specific MAC address file is there. I can see it. But no attempt is made to access it.

I different system using PXELinux BIOS as the boot type is working. I don’t know enough about the boot process to troubleshoot this further.

Thanks, really appreciate any help.

The ' character in this comment broke grub, remove it.

You say this is on 2.1? That should not be there. We found this in 2.3:

https://projects.theforeman.org/issues/31758

Yeah. I’m still fighting with it.

Can you share grub2-efi version on the TFTP smart proxy and possibly md5 sum of the grubx64.efi? There are some versions floating in RHEL/CentOS repos which are buggy.

Try with

http://people.redhat.com/~lzapleta/grub/grub2-efi-x64-2.04-35.fc34.x86_64/

This took way too long to solve, but it was my own doing.
I had another server on the network that was interfering with the traffic.