This is a tutorial how to set up UEFI HTTP Boot over tagged VLAN provisioning with Foreman 1.23 and few more patches. Hopefully all of these can be merged for Foreman 1.24, they are all small changes.
Requirements:
A host for foreman, a host for dedicated smart-proxy with connection to foreman and a managed host you want to provision. I’ve created a simple setup in libvirt where smart-proxy host has two NICs:
- Foreman VM foreman.nat.lan (NIC: 192.168.199.0)
- Smart Proxy VM gwvlan.nat.lan (NIC1: 192.168.199.0, NIC2: 192.168.55.0 with VLAN 55)
- Managed VM UEFI (NIC: 192.168.55.0 with VLAN 55)
Then I configured the managed host to boot via HTTP UEFI IPv4 boot. There is a nasty bug in UEFI firmware used in libvirt and configuration is a bit hidden, but in short: if a HTTP URL is once set, it cannot be overwritten. It will ignore all other settings, the workaround is to delete NVRAM FD image (find it in the libvirt XML) and start over. Also, for some reason it would not boot with URL in form of http://IP:PORT which was a problem because smart-proxy is running the httpboot module on port 8000. The workaround is to forward all HTTP traffic from 80 to 8000, proxy can’t actually bind reserved system ports (it’s a bug: Bug #28041: Smart proxy is unable to bind port < 1024 - Smart Proxy - Foreman).
The address for the UEFI VM firmware was: http://192.168.55.1/httpboot/grub2/grubx64.efi
If you have a switch with configurable trunking, it’s probalby easier to use it but the smart-proxy VM can act as switch/router as well. I configured interfaces and firewall to do masquerade so the traffic can go to the internet (package downloads):
nmcli c add con-name upstream ifname eth0 type ethernet ip4 192.168.199.251/24 gw4 192.168.199.1 ipv4.dns 192.168.199.1
nmcli c add con-name downstream type vlan ifname vlan55 dev eth1 id 55 ip4 192.168.55.1/24
nmcli c mod upstream connection.zone external
nmcli c mod downstream connection.zone internal
Then I opened some ports and also did the 80->8000 forward. Rich rule must be used, the native firewall-cmd forwarding will NOT work (God this was a pain to find out):
firewall-cmd --zone=external --add-port=8000/tcp --add-port=8443/tcp --permanent
firewall-cmd --zone=internal --add-port=8000/tcp --add-port=8443/tcp --permanent
firewall-cmd --zone=internal --add-service=dns --add-service=dhcp --permanent
firewall-cmd --zone=internal --add-rich-rule='rule family="ipv4" destination address="192.168.55.1" forward-port port="80" protocol="tcp" to-port="8000"' --permanent
And then the smart-proxy was installed:
foreman-installer \
--no-enable-foreman \
--no-enable-puppet \
--no-enable-foreman-cli \
--enable-foreman-proxy \
--enable-foreman-proxy-plugin-discovery \
--foreman-proxy-plugin-discovery-install-images=true \
--foreman-proxy-bind-host=\* \
--foreman-proxy-http=true \
--foreman-proxy-http-port=8000 \
--foreman-proxy-templates=true \
--foreman-proxy-templates-listen-on=both \
--foreman-proxy-template-url=http://192.168.55.1 \
--foreman-proxy-httpboot=true \
--foreman-proxy-httpboot-listen-on=both \
--foreman-proxy-tftp=true \
--foreman-proxy-tftp-servername=192.168.55.1 \
--foreman-proxy-puppet=false \
--foreman-proxy-puppetca=false \
--foreman-proxy-dhcp=true \
--foreman-proxy-dhcp-interface=vlan55 \
--foreman-proxy-dhcp-gateway=192.168.55.1 \
--foreman-proxy-dhcp-range="192.168.55.10 192.168.55.250" \
--foreman-proxy-dhcp-nameservers="192.168.55.1" \
--foreman-proxy-dns=true \
--foreman-proxy-dns-interface=vlan55 \
--foreman-proxy-dns-zone=vlan \
--foreman-proxy-dns-reverse=55.168.192.in-addr.arpa \
--foreman-proxy-dns-forwarders=192.168.199.1 \
--foreman-proxy-foreman-base-url=https://foreman.nat.lan \
--foreman-proxy-trusted-hosts=foreman.nat.lan \
--foreman-proxy-oauth-consumer-key=qzuMXcxxx \
--foreman-proxy-oauth-consumer-secret=g549SDLKjxxx
Since I haven’t installed Puppet master on the proxy, I had to generate certificate (thanks to Ewoud for the trick):
puppet ssl bootstrap --server foreman.nat.lan
puppet ca sign gwvlan.nat.lan # <<< run this on foreman
Required patches (merged):
- https://github.com/theforeman/smart-proxy/pull/670
- https://github.com/theforeman/community-templates/pull/606
- https://github.com/theforeman/community-templates/pull/636
- https://github.com/theforeman/community-templates/pull/638
- https://github.com/theforeman/foreman/pull/6983
- https://github.com/theforeman/community-templates/pull/644
- https://github.com/theforeman/community-templates/pull/641
The most important missing bit in Foreman 1.23 is that foreman_url macro is rendered with port which is used for Foreman-Proxy connection instead of the correct HTTP port (which is 8000). That is the first patch which I applied. Then it was a small patch for smart-proxy to deploy PXEGrub2 templates in expected format and two template changes in this regard (regexp module did not work in UEFI HTTP mode). Also path has to be different for UEFI HTTP so I created another template change and finally VLAN support is a PR for all template kinds: PXEGrub2, kickstart (I tested both), PXELinux and iPXE.
Due to bug in grub2 (https://bugzilla.redhat.com/show_bug.cgi?id=1763216) the global pxegrub2 template (pxegrub2_mac.erb) must be edited and absolute URL configfile statements must be commented out. These are only needed for Debian provisioning anyway. The error is: Failed to send request. status=0xf0. The global grub2 template must also be modified to point to the correct smart-proxy, port, endpoint type and VLAN must be appended: proxy.url=https://192.168.55.1:8443 proxy.type=proxy fdi.vlan.primary=55
WIP: FDI currently does not correctly initialize VLAN and fails to communicate with the other party.
When creating the host in Foreman, it be created as follows:
- NIC1: enter MAC address and identifier (
eth0), not managed, not primary and not provisioning - NIC1: enter the same MAC address, identifier
vlan55, managed, primary, provisioning, virtual, tag: 55 and attached to:eth0
That’s all. In my case the workflow was:
- UEFI VM firmware initializes IPv4
- UEFI VM firmware downloads
http://192.168.55.1/httpboot/grub2/grubx64.efi - Grub2 loads MAC-based template from HTTP server
- Anaconda kicks in with the correct VLAN dracut option
- Anaconda downloads KS via VLAN tagged network
- Anaconda finishes off the installation
Logs:
2019-10-11T08:54:47 7c2c81d7 [I] Started HEAD /httpboot/grub2/grubx64.efi
2019-10-11T08:54:47 7c2c81d7 [I] Finished HEAD /httpboot/grub2/grubx64.efi with 200 (0.53 ms)
2019-10-11T08:54:47 6446ae6c [I] Started GET /httpboot/grub2/grubx64.efi
2019-10-11T08:54:47 6446ae6c [I] Finished GET /httpboot/grub2/grubx64.efi with 200 (0.69 ms)
2019-10-11T08:54:47 a21b18d2 [I] Started HEAD /httpboot/grub2/grub.cfg
2019-10-11T08:54:47 a21b18d2 [I] Finished HEAD /httpboot/grub2/grub.cfg with 200 (0.82 ms)
2019-10-11T08:54:47 31722986 [I] Started GET /httpboot/grub2/grub.cfg
2019-10-11T08:54:47 31722986 [I] Finished GET /httpboot/grub2/grub.cfg with 200 (0.29 ms)
2019-10-11T08:54:47 c5ec3df8 [I] Started HEAD /EFI/centos/x86_64-efi/command.lst
2019-10-11T08:54:47 c5ec3df8 [E] Not found
2019-10-11T08:54:47 c5ec3df8 [I] Finished HEAD /EFI/centos/x86_64-efi/command.lst with 404 (0.35 ms)
2019-10-11T08:54:47 7d343259 [I] Started HEAD /EFI/centos/x86_64-efi/fs.lst
2019-10-11T08:54:47 7d343259 [E] Not found
2019-10-11T08:54:47 7d343259 [I] Finished HEAD /EFI/centos/x86_64-efi/fs.lst with 404 (0.34 ms)
2019-10-11T08:54:47 f9d33000 [I] Started HEAD /EFI/centos/x86_64-efi/crypto.lst
2019-10-11T08:54:47 f9d33000 [E] Not found
2019-10-11T08:54:47 f9d33000 [I] Finished HEAD /EFI/centos/x86_64-efi/crypto.lst with 404 (0.33 ms)
2019-10-11T08:54:47 b88bc9c6 [I] Started HEAD /EFI/centos/x86_64-efi/terminal.lst
2019-10-11T08:54:47 b88bc9c6 [E] Not found
2019-10-11T08:54:47 b88bc9c6 [I] Finished HEAD /EFI/centos/x86_64-efi/terminal.lst with 404 (0.37 ms)
2019-10-11T08:54:47 32a57a75 [I] Started HEAD /httpboot/grub2/grub.cfg
2019-10-11T08:54:47 32a57a75 [I] Finished HEAD /httpboot/grub2/grub.cfg with 200 (0.5 ms)
2019-10-11T08:54:47 689e636a [I] Started GET /httpboot/grub2/grub.cfg
2019-10-11T08:54:47 689e636a [I] Finished GET /httpboot/grub2/grub.cfg with 200 (0.33 ms)
2019-10-11T08:54:47 0f4b015e [I] Started HEAD /grub2/grub.cfg-52:54:00:13:13:01
2019-10-11T08:54:47 0f4b015e [I] Finished HEAD /grub2/grub.cfg-52:54:00:13:13:01 with 404 (0.22 ms)
2019-10-11T08:54:47 a4e6398e [I] Started HEAD /httpboot/grub2/grub.cfg-52:54:00:13:13:01
2019-10-11T08:54:47 a4e6398e [I] Finished HEAD /httpboot/grub2/grub.cfg-52:54:00:13:13:01 with 200 (0.8 ms)
2019-10-11T08:54:47 9a9b8132 [I] Started GET /httpboot/grub2/grub.cfg-52:54:00:13:13:01
2019-10-11T08:54:47 9a9b8132 [I] Finished GET /httpboot/grub2/grub.cfg-52:54:00:13:13:01 with 200 (0.33 ms)
2019-10-11T08:54:54 1f6bfb54 [I] Started HEAD /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-vmlinuz
2019-10-11T08:54:54 1f6bfb54 [I] Finished HEAD /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-vmlinuz with 200 (0.82 ms)
2019-10-11T08:54:54 51c1f409 [I] Started GET /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-vmlinuz
2019-10-11T08:54:54 51c1f409 [I] Finished GET /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-vmlinuz with 200 (0.31 ms)
2019-10-11T08:54:55 0755a317 [I] Started HEAD /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-initrd.img
2019-10-11T08:54:55 0755a317 [I] Finished HEAD /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-initrd.img with 200 (0.58 ms)
2019-10-11T08:54:55 76be1400 [I] Started GET /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-initrd.img
2019-10-11T08:54:55 76be1400 [I] Finished GET /httpboot/boot/centos-local-proxy-NpAPvFVIy69k-initrd.img with 200 (0.42 ms)
2019-10-11T08:55:06 91146ffa [I] Started GET /unattended/provision token=e030ac18-ffa9-4c4b-a970-8e180be67697
2019-10-11T08:55:06 681bb805 [I] Started GET /unattended/templateServer
2019-10-11T08:55:06 681bb805 [I] Finished GET /unattended/templateServer with 200 (0.31 ms)
2019-10-11T08:55:06 681bb805 [I] Started GET /unattended/templateServer
2019-10-11T08:55:06 681bb805 [I] Finished GET /unattended/templateServer with 200 (0.32 ms)
2019-10-11T08:55:06 91146ffa [I] Finished GET /unattended/provision with 200 (485.13 ms)
2019-10-11T09:15:48 cfae11de [I] Started POST /unattended/built token=e030ac18-ffa9-4c4b-a970-8e180be67697
Cheers.