UEFI HTTP Boot over tagged VLAN with discovery

This is a tutorial how to set up UEFI HTTP Boot over tagged VLAN provisioning with Foreman 1.23 and few more patches. Hopefully all of these can be merged for Foreman 1.24, they are all small changes.

Requirements:

A host for foreman, a host for dedicated smart-proxy with connection to foreman and a managed host you want to provision. I’ve created a simple setup in libvirt where smart-proxy host has two NICs:

  • Foreman VM foreman.nat.lan (NIC: 192.168.199.0)
  • Smart Proxy VM gwvlan.nat.lan (NIC1: 192.168.199.0, NIC2: 192.168.55.0 with VLAN 55)
  • Managed VM UEFI (NIC: 192.168.55.0 with VLAN 55)

Then I configured the managed host to boot via HTTP UEFI IPv4 boot. There is a nasty bug in UEFI firmware used in libvirt and configuration is a bit hidden, but in short: if a HTTP URL is once set, it cannot be overwritten. It will ignore all other settings, the workaround is to delete NVRAM FD image (find it in the libvirt XML) and start over. Also, for some reason it would not boot with URL in form of http://IP:PORT which was a problem because smart-proxy is running the httpboot module on port 8000. The workaround is to forward all HTTP traffic from 80 to 8000, proxy can’t actually bind reserved system ports (it’s a bug: Bug #28041: Smart proxy is unable to bind port < 1024 - Smart Proxy - Foreman).

The address for the UEFI VM firmware was: http://192.168.55.1/httpboot/grub2/grubx64.efi

If you have a switch with configurable trunking, it’s probalby easier to use it but the smart-proxy VM can act as switch/router as well. I configured interfaces and firewall to do masquerade so the traffic can go to the internet (package downloads):

nmcli c add con-name upstream ifname eth0 type ethernet ip4 192.168.199.251/24 gw4 192.168.199.1 ipv4.dns 192.168.199.1
nmcli c add con-name downstream type vlan ifname vlan55 dev eth1 id 55 ip4 192.168.55.1/24
nmcli c mod upstream connection.zone external
nmcli c mod downstream connection.zone internal

Then I opened some ports and also did the 80->8000 forward. Rich rule must be used, the native firewall-cmd forwarding will NOT work (God this was a pain to find out):

firewall-cmd --zone=external --add-port=8000/tcp --add-port=8443/tcp --permanent
firewall-cmd --zone=internal --add-port=8000/tcp --add-port=8443/tcp --permanent
firewall-cmd --zone=internal --add-service=dns --add-service=dhcp --permanent
firewall-cmd --zone=internal --add-rich-rule='rule family="ipv4" destination address="192.168.55.1" forward-port port="80" protocol="tcp" to-port="8000"' --permanent

And then the smart-proxy was installed:

foreman-installer \
  --no-enable-foreman \
  --no-enable-puppet \
  --no-enable-foreman-cli \
  --enable-foreman-proxy \
  --enable-foreman-proxy-plugin-discovery \
  --foreman-proxy-plugin-discovery-install-images=true \
  --foreman-proxy-bind-host=\* \
  --foreman-proxy-http=true \
  --foreman-proxy-http-port=8000 \
  --foreman-proxy-templates=true \
  --foreman-proxy-templates-listen-on=both \
  --foreman-proxy-template-url=http://192.168.55.1 \
  --foreman-proxy-httpboot=true \
  --foreman-proxy-httpboot-listen-on=both \
  --foreman-proxy-tftp=true \
  --foreman-proxy-tftp-servername=192.168.55.1 \
  --foreman-proxy-puppet=false \
  --foreman-proxy-puppetca=false \
  --foreman-proxy-dhcp=true \
  --foreman-proxy-dhcp-interface=vlan55 \
  --foreman-proxy-dhcp-gateway=192.168.55.1 \
  --foreman-proxy-dhcp-range="192.168.55.10 192.168.55.250" \
  --foreman-proxy-dhcp-nameservers="192.168.55.1" \
  --foreman-proxy-dns=true \
  --foreman-proxy-dns-interface=vlan55 \
  --foreman-proxy-dns-zone=vlan \
  --foreman-proxy-dns-reverse=55.168.192.in-addr.arpa \
  --foreman-proxy-dns-forwarders=192.168.199.1 \
  --foreman-proxy-foreman-base-url=https://foreman.nat.lan \
  --foreman-proxy-trusted-hosts=foreman.nat.lan \
  --foreman-proxy-oauth-consumer-key=qzuMXcxxx \
  --foreman-proxy-oauth-consumer-secret=g549SDLKjxxx

Since I haven’t installed Puppet master on the proxy, I had to generate certificate (thanks to Ewoud for the trick):

puppet ssl bootstrap --server foreman.nat.lan
puppet ca sign gwvlan.nat.lan # <<< run this on foreman

Required patches (merged):

The most important missing bit in Foreman 1.23 is that foreman_url macro is rendered with port which is used for Foreman-Proxy connection instead of the correct HTTP port (which is 8000). That is the first patch which I applied. Then it was a small patch for smart-proxy to deploy PXEGrub2 templates in expected format and two template changes in this regard (regexp module did not work in UEFI HTTP mode). Also path has to be different for UEFI HTTP so I created another template change and finally VLAN support is a PR for all template kinds: PXEGrub2, kickstart (I tested both), PXELinux and iPXE.

Due to bug in grub2 (https://bugzilla.redhat.com/show_bug.cgi?id=1763216) the global pxegrub2 template (pxegrub2_mac.erb) must be edited and absolute URL configfile statements must be commented out. These are only needed for Debian provisioning anyway. The error is: Failed to send request. status=0xf0. The global grub2 template must also be modified to point to the correct smart-proxy, port, endpoint type and VLAN must be appended: proxy.url=https://192.168.55.1:8443 proxy.type=proxy fdi.vlan.primary=55

WIP: FDI currently does not correctly initialize VLAN and fails to communicate with the other party.

When creating the host in Foreman, it be created as follows:

  • NIC1: enter MAC address and identifier (eth0), not managed, not primary and not provisioning
  • NIC1: enter the same MAC address, identifier vlan55, managed, primary, provisioning, virtual, tag: 55 and attached to: eth0

That’s all. In my case the workflow was:

  • UEFI VM firmware initializes IPv4
  • UEFI VM firmware downloads http://192.168.55.1/httpboot/grub2/grubx64.efi
  • Grub2 loads MAC-based template from HTTP server
  • Anaconda kicks in with the correct VLAN dracut option
  • Anaconda downloads KS via VLAN tagged network
  • Anaconda finishes off the installation

Logs:

2019-10-11T08:54:47 7c2c81d7 [I] Started HEAD /httpboot/grub2/grubx64.efi 
2019-10-11T08:54:47 7c2c81d7 [I] Finished HEAD /httpboot/grub2/grubx64.efi with 200 (0.53 ms)
2019-10-11T08:54:47 6446ae6c [I] Started GET /httpboot/grub2/grubx64.efi 
2019-10-11T08:54:47 6446ae6c [I] Finished GET /httpboot/grub2/grubx64.efi with 200 (0.69 ms)
2019-10-11T08:54:47 a21b18d2 [I] Started HEAD /httpboot/grub2/grub.cfg 
2019-10-11T08:54:47 a21b18d2 [I] Finished HEAD /httpboot/grub2/grub.cfg with 200 (0.82 ms)
2019-10-11T08:54:47 31722986 [I] Started GET /httpboot/grub2/grub.cfg 
2019-10-11T08:54:47 31722986 [I] Finished GET /httpboot/grub2/grub.cfg with 200 (0.29 ms)
2019-10-11T08:54:47 c5ec3df8 [I] Started HEAD /EFI/centos/x86_64-efi/command.lst 
2019-10-11T08:54:47 c5ec3df8 [E] Not found
2019-10-11T08:54:47 c5ec3df8 [I] Finished HEAD /EFI/centos/x86_64-efi/command.lst with 404 (0.35 ms)
2019-10-11T08:54:47 7d343259 [I] Started HEAD /EFI/centos/x86_64-efi/fs.lst 
2019-10-11T08:54:47 7d343259 [E] Not found
2019-10-11T08:54:47 7d343259 [I] Finished HEAD /EFI/centos/x86_64-efi/fs.lst with 404 (0.34 ms)
2019-10-11T08:54:47 f9d33000 [I] Started HEAD /EFI/centos/x86_64-efi/crypto.lst 
2019-10-11T08:54:47 f9d33000 [E] Not found
2019-10-11T08:54:47 f9d33000 [I] Finished HEAD /EFI/centos/x86_64-efi/crypto.lst with 404 (0.33 ms)
2019-10-11T08:54:47 b88bc9c6 [I] Started HEAD /EFI/centos/x86_64-efi/terminal.lst 
2019-10-11T08:54:47 b88bc9c6 [E] Not found
2019-10-11T08:54:47 b88bc9c6 [I] Finished HEAD /EFI/centos/x86_64-efi/terminal.lst with 404 (0.37 ms)
2019-10-11T08:54:47 32a57a75 [I] Started HEAD /httpboot/grub2/grub.cfg 
2019-10-11T08:54:47 32a57a75 [I] Finished HEAD /httpboot/grub2/grub.cfg with 200 (0.5 ms)
2019-10-11T08:54:47 689e636a [I] Started GET /httpboot/grub2/grub.cfg 
2019-10-11T08:54:47 689e636a [I] Finished GET /httpboot/grub2/grub.cfg with 200 (0.33 ms)
2019-10-11T08:54:47 0f4b015e [I] Started HEAD /grub2/grub.cfg-52:54:00:13:13:01 
2019-10-11T08:54:47 0f4b015e [I] Finished HEAD /grub2/grub.cfg-52:54:00:13:13:01 with 404 (0.22 ms)
2019-10-11T08:54:47 a4e6398e [I] Started HEAD /httpboot/grub2/grub.cfg-52:54:00:13:13:01 
2019-10-11T08:54:47 a4e6398e [I] Finished HEAD /httpboot/grub2/grub.cfg-52:54:00:13:13:01 with 200 (0.8 ms)
2019-10-11T08:54:47 9a9b8132 [I] Started GET /httpboot/grub2/grub.cfg-52:54:00:13:13:01 
2019-10-11T08:54:47 9a9b8132 [I] Finished GET /httpboot/grub2/grub.cfg-52:54:00:13:13:01 with 200 (0.33 ms)
2019-10-11T08:54:54 1f6bfb54 [I] Started HEAD /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-vmlinuz 
2019-10-11T08:54:54 1f6bfb54 [I] Finished HEAD /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-vmlinuz with 200 (0.82 ms)
2019-10-11T08:54:54 51c1f409 [I] Started GET /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-vmlinuz 
2019-10-11T08:54:54 51c1f409 [I] Finished GET /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-vmlinuz with 200 (0.31 ms)
2019-10-11T08:54:55 0755a317 [I] Started HEAD /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-initrd.img 
2019-10-11T08:54:55 0755a317 [I] Finished HEAD /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-initrd.img with 200 (0.58 ms)
2019-10-11T08:54:55 76be1400 [I] Started GET /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-initrd.img 
2019-10-11T08:54:55 76be1400 [I] Finished GET /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-initrd.img with 200 (0.42 ms)
2019-10-11T08:55:06 91146ffa [I] Started GET /unattended/provision token=e030ac18-ffa9-4c4b-a970-8e180be67697
2019-10-11T08:55:06 681bb805 [I] Started GET /unattended/templateServer 
2019-10-11T08:55:06 681bb805 [I] Finished GET /unattended/templateServer with 200 (0.31 ms)
2019-10-11T08:55:06 681bb805 [I] Started GET /unattended/templateServer 
2019-10-11T08:55:06 681bb805 [I] Finished GET /unattended/templateServer with 200 (0.32 ms)
2019-10-11T08:55:06 91146ffa [I] Finished GET /unattended/provision with 200 (485.13 ms)
2019-10-11T09:15:48 cfae11de [I] Started POST /unattended/built token=e030ac18-ffa9-4c4b-a970-8e180be67697

Cheers.

2 Likes

I wonder if we should add this to:
https://theforeman.org/manuals/latest/index.html#3.2.3InstallationScenarios

This should work by default.

Should also be default

This is the default so redundant.

This is only needed for situations where the host doesn’t exist yet (like discovery)

I’d suggest using something like vlan.example.com

But it did not, I was troubleshooting this for a while until I realized the listen option had ;; in it. This fixed it.

I can confirm, I wanted to explicitly show that however because I am configuring firewall port fordwarding.

Not sure about this one, it was either this or templates module what was listenint only on HTTP and not on HTTPS.

I agree with the rest.