Deploying ESXi 6.0/6.5 on UEFI machines

Problem:
I need to install ESXi 6.0/6.5 on UEFI machines. But I only find BIOS/PXELinux Deployment-HowTo in the internet. Can we build one here?

Expected outcome:
I expect to handle a ESXi installation in a similar fashion like any other Linux (RHEL, CentOS, Debian, …) - expect for the Puppet / Configuration part.

Foreman and Proxy versions:
Foreman 1.17.0

Other relevant data:
I know this:

I have setup my DHCP to deliver different configs based on the system architecture. When a UEFI system is booting it will get an offer with the path to /grub2/grubx64.efi . As for now I can deploy CentOS and RHEL with this setup. :wink: but it seems that I miss something (knowledge wise) to get ESXi in combination with UEFI working.
I thought it is similar to CentOS for the first few steps… but… well no… or I miss more than I can oversee right know. :wink:

Other Facts:
install media is on a second server: ftp://storage.local/pub/isos/esxi/$major/$minor/$arch/
e.g.: ftp://storage.local/pub/isos/esxi/6/0/x86_64/

also a good source: https://www.vmware.com/techpapers/2015/installing-vmware-esxi-6.0-using-pxe-10508.html
starting from page 10 - Point 3)

Did you ever figure this out? I am trying to work it out and keep getting “invalid magic number” when trying to boot the esxi installation via PXE (in UEFI)

My config ends up with this entry:
linuxefi /os/esxi/efi/boot/bootx64.efi -c /os/esxi/efi/boot/boot.cfg

I have edited the boot.cfg as the documentation states, removeing the slashs in the modules line, removing the “cdrom” part of “kernelopt” and adding a prefix line with the root path to the installation files.

I think my problem is specifically with the grub2 syntax for installing esxi which as you pointed out there are basically no instructions for online.

If you got this figured out I would love to know how.

I am able to load the ESXi “installer” with a copy fo the Default PXEGrub2 template. But it still not works to the end… maybe someone can help.

I used the template “Kickstart default PXEGrub2” and changed it just a little bit.

I added a line to specify my cfg-folder:

esxiCFGfolder = '01-' + mac.gsub(':', '-')

At the very end I replaced the menu with this:

menuentry ‘<%= template_name %>’ {
insmod chain
chainloader /esxi/6/0/x86_64/efi/boot/bootx64.efi -c /<%= esxiCFGfolder %>/boot.cfg
}

The example output would look like this:

# This file was deployed via ‘PXEGrub2 chainload ESXi’ template

set default=0 set timeout=3 menuentry 'PXEGrub2 chainload ESXi' { insmod chain chainloader /esxi/6/0/x86_64/efi/boot/bootx64.efi -c /01-00-50-56-86-5e-89/boot.cfg }

When I use “linuxefi” instead of “chainloader” I get the same issue as reported above.

All this is based on the following layout: [TFTP folder]

[admin@FOREMAN tftpboot]$ tree -L 2 .
tree output

The cfg-file of ESXi [copy of efi/boot/boot.cfg]

[admin@FOREMAN tftpboot]$ cat 01-00-50-56-86-5e-89/boot.cfg
ESXi boot.cfg file

When I now fetch the TFTP logs it shows, that all files are fetched except for the cfg-file:

[admin@FOREMAN tftpboot]$ sudo tail -f /var/log/messages | grep tftp
Output of /var/log/Messages for TFTP

I m getting “Fatal error: 20 (not stared)”.

Does someone have a guess why the cfg-file is not fetched? Or/And what we miss here?

Do you mean “invalid magic number” like I posted or something you mentioned earlier?

When I use chainloader I get the same error as you, except error 15 (not found) (first line about parsing is the same for me).

I have also tried the boot.cfg via http and ftp and the same result.

As an aside, my BIOS boot works just fine for ESXi via syslinux. I have double/triple checked permissions, ownerships, etc.

I am working with a colleague who is far more proficient at Foreman than I am, he thinks its an issue with the boot loader(s). I tend to agree as using grub2 with linuxefi doesnt even get to the installer but chainloading the ESXi bootloader at least gets into the installation screen.

Also, have you modified your boot.cfg, taking out the slashes, adding prefix line, removing the cdrom portion of the kernelopt line? As a side note, I have also tried renaming bootx64.efi to mboot.efi and moving the .efi and boot.cfg files to the root of tftpboot as per ESXi documenation…no change (if anything worse).

I did also notice in the official documentation here for ESXi PXE that they specifically mention using iPXE for UEFI, which I can confirm works (though I have not tried adding/using iPXE under Foreman). I was able to UEFI boot the installer via iPXE on another system.

So I think our options are try and get grub2 and the ESXi bootloader and config to play nice, or leverage iPXE within Foreman. I haven’t found any decent documentation to explain how to use iPXE in Foreman however.

I will of course report back any findings, hopefully if you solve it you will share too as I have been at this for days, maybe longer and have little hair left to pull out. Thanks

https://communities.vmware.com/thread/463818?start=15&tstart=0

This post seems to indicate its a issue with grub2. The one person from VMWare claims to have put a work around into the bootx64.efi bootloader to compensate for the grub2 issue.

I thought based on their posts that using the bootx64.efi (mboot.efi) from the newer ESXi 6.7 might help. I copied it to the same directory as the 6.5 version of bootx64.efi and adjusted my template, confirming the change in grub upon boot. I still get the same error when chainloading.

I think the basic gist is, VMWare only officially supports using their bootloader or using iPXE. Grub2 appears to have some sort of flaw preventing it from pxe booting ESXi.

Some off the wall thinking, not knowing even a fraction of what there is to know about Foreman, but…

Is it possible to somehow have an entry for unknown hosts that reboots the unknown host and doesnt load the grub2 bootloaders but instead forces loading the ESXi bootloader?

Something like: grub2 -> menu entry “Install ESXi” -> reboot -> DHCP’s bootloader assignment skipped -> default menu skipped -> ESXi bootloader loaded -> ESXi installation.

Maybe some flag temporarily stored server side for the specific unknown host to prevent the standard bootloaders from being used that instance?

well… technically I get it… but… I am not sure/convinced that this is still true. Because we are able to load the installer, right? so the issue of grub2 is solved (?) in the meantime I guess. On the other hand it would explain why the boot.cfg file is not loaded… My thoughts:

  1. the GRUB2 is fetched (EFI on)
  2. chainload of next boot loader (EFI on)
  3. some kind of kexec (EFI goes off)
  4. the next boot loader starts (EFI off) but can’t load more files because EFI-network is off/down…?!?

everything I did so far, was with ESXi 6.0 U3 (file: VMware-ESXi-6.0.0-Update3-5050593-HPE-600.9.7.0.17-Feb2017.iso). So I spend some time to load the other ISOs to my TFTP to use a more up to date version of the efi-loaders. I always read that the new boot loaders a backward compatible, so I thought I’ll give it a try. But now, I get a new error: Permission denied

the behaviour on the TFTP is the same:

Output of /var/log/Messages of TFTP for ESXi 6.5

What I miss is the last line like:

........Client 192.168.0.121 finished /esxi/6/5/x86_64/efi/boot/bootx64.efi

Is this maybe a hint?

On which server? The DHCP or Foreman?

I hope, that we can do everything on the Foreman server. Because I don’t want to tough and configure the DHCP all the time - even with scripts. Technically it is possible to script a change on the DHCP for a specific MAC to fetch a specific file. But since we build an destroy the target host on an daily bases I would like to keep the DHCP config more generic/static.

DHCP > TFM GRUB2 > to-what-ever-next-loader

Because the change of the TFM GRUB2 is already scripted and automated. :wink:

OK, so I did a bit troubleshooting. I changed the grub2-config-file to:

  menuentry 'PXEGrub2 chainload ESXi 6.0 x64' { insmod chain chainloader /grub2/shimx64.efi }

This works. I can chainload another efi bootloader. the TFTP log shows that shimx64.efi is fetched. This tells me, that even after “leaving” GRUB2 we still can fetch data over the network… so question for me is, why is the esxi-loader not able to load it.

supplement:

when I use the setup for ESXi 6.0 I can see, that I get the confirmation, that the file was fetched successfully:

May 9 15:34:45 FOREMAN in.tftpd[20350]: RRQ from 192.168.0.121 filename /grub2/grub.cfg-01-00-50-56-86-5e-89 May 9 15:34:45 FOREMAN in.tftpd[20350]: Client 192.168.0.121 finished /grub2/grub.cfg-01-00-50-56-86-5e-89 May 9 15:34:48 FOREMAN in.tftpd[20353]: RRQ from 192.168.0.121 filename /esxi/6/0/x86_64/efi/boot/bootx64.efi May 9 15:34:48 FOREMAN in.tftpd[20353]: Client 192.168.0.121 finished /esxi/6/0/x86_64/efi/boot/bootx64.efi

I was just posing this as an idea, I do not know if its technically possible or feasible.

As you are finding, and my result pretty much mirror, its possible for grub2 to chainload another bootloader which can then function normally. It is something specific to how grub2 chainloads the ESXi bootloader that is the issue.

As you mention, newer versions should work backwards, but I tried the latest ESXi 6.7 bootloader to the same effect. So any workaround VMWare put into it either was permanent or no longer resolves the issue, while on the other hand grub2 has not addressed any issue on their end.

I think ESXi’s bootloader is problematic overall. The approach to PXE booting it is very different then dealing with Linux distros, Windows or utilities that are bootable. ESXi specifically has been problematic for me in every attempt to UEFI PXE boot it that I have made. I can get SYSLINUX to boot it BIOS just fine.

As per VMWare documentation and my extensive testing I have found only iPXE able to UEFI PXE boot ESXi 6.x. Even at that it was problematic on 3 different R620’s (PSOD) but worked fine on an FC630.

Between Syslinux, Grub/2 and iPXE…iPXE has been the only bootloader I have found that seems to work well under UEFI boot. It has its issues, but Syslinux development is pretty much dead as far as I can tell (last official release is a few years old) and I feel like Grub/2 can be used for PXE booting but was not solely designed for it and is not developed with PXE at the forefront of its development.

I am hoping that Foreman either already supports iPXE in a capacity that would help you and I or will soon. I have posted here asking about the matter of using iPXE for all unknown hosts so that we can leverage it.

I am currently exploring other approaches like a manually setup PXE server using iPXE. At this rate the options for UEFI PXE booting ESXi are fairly limited but the alternative solutions for imaging/deployment are not as limited.

Hello there,

Like you I too have been looking for a solution to this for a long time. Thanks to Lukas, I finally managed to get a fully working setup! Please have a look at this thread for details.

2 Likes