Foreman handles PXE provisioning by deploying PXELinux and/or Grub2 configuration files to a TFTP server using a smart-proxy. Templates of PXELinux and PXEGrub2 kinds are used to generate configuration entries which either boots into OS installer when a host is in build mode, or skips booting from network when a host is not in build mode. This works for almost a decade now.
But there are cases when PXE cannot be used - for example on networks with unmanaged DHCP servers, PXE service blacklisted or firewalled or simply when TFTP UDP-based protocol is not reliable, for example on 3G/4G networks as we have users doing provisioning in moving vehicles, trains and ships. In this case, two possible solutions are availble: iPXE and HTTP UEFI Boot. This article covers the former.
Before proceeding with iPXE configuration, there is one important thing to keep in mind: iPXE does not use PXE and therefore supported hardware is limited. We ocassionally see bare-metal servers not being able to recognize network cards or error out during transfer. This also happened with some hypervisors, so make sure to test first before ordering more hardware or paying licenses or subscriptions.
The scenario is simple, we want to configure Foreman to chainboot iPXE over PXE, then iPXE takes over and fetches provisioning script which either loads OS installer or skips to load the next entry in the boot order (which is expected to be the installed OS). In this scenario, we still leverage PXE/TFTP to download ipxe.efi file which is approximately 700kB. Then all communication is HTTP.
Everything was tested on Red Hat Enterprise Linux 7.6 with Foreman 1.20 and Satellite 6.5. The first step is to copy ipxe.efi to the TFTP directory. Letâs also grab undionly.kpxe file for BIOS systems.
cp /usr/share/ipxe/undionly.kpxe /var/lib/tftpboot/undionly.0
cp /usr/share/ipxe/ipxe.efi /var/lib/tftpboot/
cp /usr/share/ipxe/ipxe.lkrn /var/lib/tftpboot/
The third file called ipxe.lkrn is not used in this tutorial, but it is possible to load iPXE from PXELinux like a linux kernel (e.g. using KERNEL statement). An example is in âPXELinux chain iPXEâ tempate, just in case PXELinux must be preserved for existing systems to PXE boot.
Please keep in mind that iPXE binary in RHEL is built without security features, therefore HTTPS cannot be used and only HTTP. This is intentional and all security-related features of iPXE in Red Hat Enterprise Linux are not supported. In case you want to use HTTPS or in case hardware driver is not yet present in the iPXE from RHEL, download the up-to-date build.
This article describes discovery workflow, itâs good idea to put foreman discovery image as close to managed nodes. For this reason, we will leverage new smart-proxy module called HTTPBoot to serve FDI image (which is about 250-300 MB in size). The feature currently cannot be enabled via the installer, but manual change in the config file works fine:
/etc/foreman-proxy/settings.d/httpboot.yml
:enabled: true
Just restart foreman-proxy and refresh features to get the HTTPBoot feature appear in the list of features. Repeat for all proxies as needed.
Foreman 1.20+ comes with a template called âKickstart default iPXEâ or âPreseed default iPXEâ, thatâs the one we are going to use to load OS installer. Create an Operating system, associate the iPXE template with it and make sure the PXE Loader is set to âNoneâ, this is very important. Optionally create a host group.
To leverage the new feature of Foreman 1.20 called iPXE bootstrapping, letâs configure dhcpd.conf as follows:
if exists user-class and option user-class = "iPXE" {
filename "http://foreman.nat.lan/unattended/iPXE?bootstrap=1";
} elsif option architecture = 00:06 {
filename "ipxe.efi";
} elsif option architecture = 00:07 {
filename "ipxe.efi";
} elsif option architecture = 00:09 {
filename "ipxe.efi";
} else {
filename "undionly.0";
}
The filename option for iPXE class can also be a smart-proxy with templates module enabled, in that case the port needs to be changed. For example âhttp://capsule1.nat.lan:8000/unattended/iPXE?bootstrap=1â, typical port defaults are 8448 for foreman installation scenario and 8000 for katello installation scenario.
When EFI host boots from PXE, itâs handed over with ipxe.efi. BIOS systems are handed over with undionly.kpxe. In both cases, iPXE loads, configures network via DHCP and gets filename option with URL ending â/unattended/iPXEâ. Thatâs the moment when the new feature kicks in - when Foreman 1.20 sees the bootstrap option, it proceeds as follows:
- If the remote IP address is a known host from inventory which is in build mode, hand over its iPXE provisioning template which loads the OS installer.
- If the IP is not known, return template called âiPXE global defaultâ similarly to PXELinux or Grub2 workflows.
So itâs the IP address that makes the difference, this means that DHCP server must be under Foreman control in order to get this scenario working. But with upcoming Foreman 1.21 this actually changes as intermediate template is being shipped. In that case, Foreman proceeds as follows:
- If there is bootstrap flag set, hand over âiPXE intermediate scriptâ specified in Administer - Settings - Provisioning.
- If the remote IP address is a known host from inventory which is in build mode, hand over its iPXE provisioning template which loads the OS installer.
- If the IP is not known, return template called âiPXE global defaultâ similarly to PXELinux or Grub2 workflows.
The workflow is not complete and it does not need a managed DHCP server because the intermediate script tries to re-download the iPXE template but this time with MAC address as a HTTP parameter so the host in the inventory is matched not by a remote IP but with its MAC address. As long as the MAC address is correct in the inventory, it will work. The intermediate script basically tries all available network cards until Foreman reports HTTP 200.
So to recap, Foreman 1.20 can do iPXE with managed DHCP server, Foreman 1.21 (not yet released at the time of writing) will work even without managed DHCP servers.
To boot discovery, template named âiPXE global defaultâ needs to be modified to use HTTPBoot proxy endpoint. There is a pull request to add those as menu items so no changes will be required in future versions. Here are the relevant lines:
dhcp
kernel http://${next-server}:8000/httpboot/boot/fdi-image/vmlinuz0 initrd=initrd0.img rootflags=loop root=live:/fdi.iso rootfstype=auto ro rd.live.image acpi=force rd.luks=0 rd.md=0 rd.dm=0 rd.lvm=0 rd.bootif=0 rd.neednet=0 nomodeset proxy.url=<%= foreman_server_url %> proxy.type=foreman BOOTIF=01-${net0/mac}
initrd http://${next-server}:8000/httpboot/boot/fdi-image/initrd0.img
boot
Note that âinitrd=initrd0.imgâ option must be present, otherwise discovery will fail with kernel panic when booted in EFI mode.
Foreman workflow is based on setting all managed servers to boot from network and then from HDD, therefore Foreman can decide if to reinstall OS or exit PXE and boot from secondary device (HDD). But in EFI world, this does not work reliably - OS installer usually creates a new EFI entry as the first, so after restart the server boots into the new system. The issue is reprovisioning - when the host is put into build mode in Foreman and the server is restarted, it will ignore this and boot into the old system.
The solution is to modify EFI boot order back to boot from previous entry, assuming that the first entry was PXE/iPXE. There is a proposal snippet included from provisioning template, but since this does not work reliably (e.g. in libvirt this fails) the proposed solution is opt-it only via âefi_bootentryâ host parameter:
- When set to âpreviousâ after OS installation the new entry will be removed from top and appended as the last entry.
- When set to arbitrary text, this will be used as input to âgrep -Eâ as a regular expression to find boot entry to set as the first. Every EFI firmware vendor names PXE/HTTP Boot entries differently, so there is no standard and a regular expression for all hosts must be created unfortunately.
- No action is taken when not set (host will boot from HDD everytime).
Hopefully this post puts some light to iPXE provisioning in EFI mode. This workflow can be used with or without discovery phase. For the latter, just create host entries in Foreman via UI, API or CLI with correct MAC address.
Also it is worth noting that some hypervisors like KVM/QEMU, oVirt or Red Hat Enterprise Virtualization already comes with iPXE firmware. Others like VMWare can be customized to have iPXE as the primary PXE stack as well. This workflow works in these cases, just the PXE part of loading ipxe.efi is completely skipped as itâs not necessary and the communication is all HTTP.
Relevant reading: