ESXi Deployment via UEFI - Hook Scripts Issue

foreman-hooks
vmware
provisioning

#1

Hi,

I’ve installed Foreman 1.19.1 (on ubuntu server 16.04) with the following enabled:

–enable-foreman --enable-foreman-cli --enable-foreman-proxy --enable-puppet
–enable-foreman-plugin-dhcp-browser --enable-foreman-plugin-discovery --enable-foreman-plugin-hooks --enable-foreman-plugin-tasks --enable-foreman-plugin-templates
–enable-foreman-proxy-plugin-dhcp-remote-isc --enable-foreman-proxy-plugin-discovery --enable-foreman-proxy-plugin-dynflow

DHCP has been enabled using the foreman-installer with the isc as the provider. Then followed the following guide (only configuring UEFI):

https://theforeman.org/2018/08/deploying-esxi-through-foreman.html

When I create a host and enabled build mode, the 01-{mac:address} folder with the corresponding boot.cfg file is not created. When booting the host, it fails after getting a DHCP lease:

Start PXE over IPv4
Station IP address is 10.11.12.50
Server IP address is 10.11.12.4
NBP filename is grub2/bootx64.efi
NBP filesize is 0 bytes
PXE-E23: Client received TFTP error from server.

The host does not load mboot.efi. The contents of a Dell esxi 6.5 build 8935087 iso have been copied to /var/lib/tftpboot/boot/ESXi-6.5-8935087 and its corresponding mboot.efi copied to /var/lib/tftpboot (ownership to foreman-proxy:root)

The scripts have been discovered:

2018-11-13T14:00:11 [I|app|] Finished discovering 2 hooks for Host::Managed#after_build
2018-11-13T14:00:11 [I|app|] Finished discovering 2 hooks for Host::Managed#before_provision

The after_build and before_provisions logs keep complaining jgrep is not found:

/usr/share/foreman/config/hooks/host/managed/after_build/hook_functions.sh: line 24: jgrep: command not found
/usr/share/foreman/config/hooks/host/managed/before_provision/hook_functions.sh: line 24: jgrep: command not found

I’ve gone through the config multiple times and it looks exactly like what the guide suggested without any deviations.

Any help to point me in the right direction would be much appreciated.

Thanks.


#2

Perhaps installing tfm-rubygem-jgrep.noarch will resolve that issue.


#3

I was about to install jgrep but then found the binary already exists at:

-rwxr-xr-x 1 foreman foreman 494 Nov 1 13:08 /usr/share/foreman/vendor/ruby/2.3.0/bin/jgrep*
-rwxrwxr-x 1 foreman foreman 4609 Nov 1 13:08 /usr/share/foreman/vendor/ruby/2.3.0/gems/jgrep-1.4.1/bin/jgrep*

The “after” and “before” hook scripts source the hook_functions.sh script at the very start like so:

. $(dirname $0)/hook_functions.sh

Is there something that needs to be done to sort of “enable” jgrep to be functional?

I read somewhere to run the following:

gem install jgrep

Not sure if that’s necessary but I’m still finding my way around this “gem” stuff.

Thanks.


#4

I’m not sure why you’re encountering that particular error, but you can probably workaround it without altering the script by adding /usr/share/foreman/vendor/ruby/2.3.0/bin/ to the PATH of whatever user is invoking that script. This is usually done in .bashrc.


#5

Dug deeper after some googling and ran the command …

gem list

… and jgrep and foreman_hooks were not listed. Noticed the following bundle at …

cat /usr/share/foreman/bundler.d/foreman_hooks.rb
gem ‘foreman_hooks’, ‘0.3.14’
gem ‘jgrep’, ‘1.4.1’

… which corresponds to the versions located at /usr/share/foreman/vendor/ruby/2.3.0/gems. So after more googling, ran the following command …

cd ~foreman && sudo -u foreman bundle install

… but ‘gem list’ still did not show jgrep or foreman_hooks as installed. The Gemfile has this line …

Dir["#{File.dirname(FOREMAN_GEMFILE)}/bundler.d/*.rb"].each do |bundle|
self.instance_eval(Bundler.read_file(bundle))

… which I assume means include all .rb files under the bundler.d directory.

Must be something simple I’m missing to get the 2 gems jgrep and foreman_hooks installed.

Thanks.


#6

I’ve got past the jgrep not found issue. Had to install the following gems:

gem install /usr/share/foreman/vendor/cache/jgrep-1.4.1.gem --local --minimal-deps --verbose
gem install /usr/share/foreman/vendor/cache/foreman_hooks-0.3.14.gem --local --minimal-deps --verbose

/tmp/after_build.log and /tmp/before_provision.log now have no errors:

Thu Nov 15 10:04:21 AWST 2018: received after_build on esxi65build04
Thu Nov 15 10:08:32 AWST 2018: received before_provision on esxi65build04

But still failing when PXE booting with the same PXE error:

NBP filename is grub2/bootx64.efi
NBP filesize is 0 bytes
PXE-E23: Client received TFTP error from server

Its not handing over to mboot.efi and I also noticed when the host is enabled for build mode, the '01-{mac-address} directory is not created in /var/lib/tftpboot.

I’ve given others write permission to the tftpboot directory but still no joy;

drwxr-xrwx 9 root nogroup 4096 Nov 15 10:03 tftpboot/

Without the ‘01-{mac-address}’ directory and the boot.cfg file inside it, I’m assuming the PXE boot doesn’t handover to mboot.efi.

Any help would be much appreciated. Thanks.


#7

Progress - somewhat.

This is not valid anymore in 1.19.1 from the documentation (Foreman :: Deploying ESXi through Foreman):

$(hook_data host.host.name) # Yes it is host.host.<attribute> instead of host.<attribute> due to this open issue: https://github.com/theforeman/foreman_hooks/issues/46

Once the 2nd “host.” was removed, the hook scripts worked and did create the 01-{mac-address} directory with the corresponding boot.cfg file in it.

But I’m still at the same stumbling block where its trying to boot to the wrong boot file:

NBP filename is grub2/bootx64.efi
NBP filesize is 0 bytes

dhcpd.conf is as per the documention with respect to this segment:

#Bootfile Handoff
next-server 127.0.0.1;
option architecture code 93 = unsigned integer 16 ;
if option architecture = 00:06 {
filename “grub2/bootia32.efi”;
} elsif option architecture = 00:07 {
filename “grub2/bootx64.efi”;
} elsif option architecture = 00:09 {
filename “grub2/bootx64.efi”;
} else {
filename “pxelinux.0”;
}

DHCP lease for the machine is as such:

lease 10.11.12.50 {
starts 5 2018/11/16 07:11:08;
ends 5 2018/11/16 19:11:08;
tstp 5 2018/11/16 19:11:08;
cltt 5 2018/11/16 07:11:08;
binding state active;
next binding state free;
rewind binding state free;
hardware ethernet a0:36:9f:27:be:00;
set vendor-class-identifier = “PXEClient:Arch:00007:UNDI:003016”;
}

So its not getting to mboot.efi. The contents of the grub2 directory is as such:

lrwxrwxrwx 1 root root 7 Nov 1 13:13 boot -> …/boot/
-rw-r–r-- 1 foreman-proxy root 1405 Nov 1 13:13 grub.cfg
-rw-r–r-- 1 root root 1182720 Nov 1 13:13 grubx64.efi
lrwxrwxrwx 1 root root 11 Nov 1 13:13 shim.efi -> grubx64.efi

There’s no bootx64.efi in there.

I’m missing something here.

Thanks.


#8

Hello,
The mboot.efi bootloader does not exist by default in Foreman. You need to edit the source code to create an entry for it. It’s under section 3.2 of the guide. Once that is done, change the PXE Loader of the host to this newly created entry and set build mode to on. Then in dhcpd.leases file you should see an entry of the host with bootfile as mboot.efi.
grub2 does not even come into the picture here because it cannot chainload the ESXi EFI bootloader.


#9

Hi Amardeep,

I’ve followed the guide exactly except for section 2 which is BIOS boot which I don’t need. The entire section 3 was completed, including 3.2.

Once build mode is enabled, I verify that after_build.log is updated and I then PXE boot the physical machine. But it fails to load mboot.efi. As shown earlier, the dhcp lease does not have the bootfile entry.

dhcpd.conf does have the bootfile handoff section like in the requirements section of the guide. But how does this part …

dhcpd.leases file you should see an entry of the host with bootfile as mboot.efi

… occur? Is there some other configuration that will make this happen?

Thanks.


#10

If the host PXE-Loader is anything other than “None”, Foreman creates an entry for that host in the leases file with the appropriate pxe loader as bootfile once build mode is set. No additional configuration is needed.
A couple of stuff to verify for you:

  • Try to deploy any other Linux OS (CentOS) through Foreman. See if the lease file points to the right pxe loader once build mode is set.
  • When you set the custom PXE loader for mboot.efi in the host on the webpage, does Foreman throw an error?

#11

Foreman doesn’t throw any error after setting the PXE Loader for the host on the web page.

I’ll test things out again on Monday when I get back into the office.

Thanks.


#12

Success!

Here are the changes I needed to make to get it working:

  1. As I’m using version 1.19.1, I had to modify the hook scripts and change “host.host.” to just “host.”
  2. The after_build hook script needed to cd /var/lib/tftpboot, else it could not read the boot.cfg file to copy. Or just use the full path to copy.
  3. The after_build hook script needed to add the location of the kickstart script as such:

puppet_proxy_name=$(hook_data host.puppet_proxy_name)
puppet_token=$(hook_data host.token)
sed -e “/kernelopt=cdromBoot runweasel/c\kernelopt=cdromBoot runweasel ks=https://${puppet_proxy_name}/unattended/provision?token=${puppet_token}” -i $macdir/boot.cfg

  1. The foreman user needed write access to /var/lib/tftpboot
  2. A subnet needed to be created on the Foreman web interface for the DHCP scope (DOH!)

Then create a host in build mode as per the doco, configure the host for uefi boot and the nic as the first boot sequence, disk as second boot sequence. The host will then pick up a DHCP lease, load mboot.efi and run the installer. After installation, the host will reboot, fail to pxe boot and then boot to the next option which is the disk where esxi is installed.

Cheers.


#13

I highly suggest to use Python for hooks, we have an example script there. Both bash and Ruby are a bit problematic (shell scripting is usually a bit limited, jgrep does not work well with Ruby SCL and you need to reinstall it after upgrade manually).


#14

Hi Lukasz,
I have updated the blogpost document with the issues that surfaced in this thread. Please review the PR.


#15

Thank you very much for bringing these issues to light. I have filed a PR to update the document with these changes.