PXE-E32 Open Timeout (but TFTP works for newly provisioned VMs)

Support
#1

Problem:
Hosts provisioned by a previous Foreman server experience “PXE-E32 Open timeout”, and get stuck - eventually skipping TFTP after approx 15 minutes.
Hosts provisioned by current Foreman server are booting fine.

Expected outcome:
Hosts not provisioned by Foreman should be able to boot normally.

Foreman and Proxy versions:
Foreman proxy 1.24.2 (Features: Pulp, Templates, DHCP, Puppet CA, Puppet, Logs, HTTPBoot, and TFTP)

Foreman and Proxy plugin versions:
Foreman-tasks 0.17.5
Katello 3.14.1

Distribution and version:
Oracle Linux 7

Other relevant data:

This Foreman/katello server was built to replace a previous one which has been shut down. Provisioning new VMs is working, but hosts created with the previous server are experiencing this TFTP timeout issue when they boot.

This occurs immediately after DHCP.
/var/log/messages indicates client successfully getting an IP address (.72) from the new foreman server (.48)
Feb 24 15:49:18 foreman-new dhcpd: DHCPDISCOVER from 00:50:56:bc:fe:ea via ens192
Feb 24 15:49:18 foreman-new dhcpd: DHCPOFFER on (redacted).72 to 00:50:56:bc:fe:ea via ens192
Feb 24 15:49:20 foreman-new dhcpd: DHCPREQUEST for (redacted).72 ((redacted).48) from 00:50:56:bc:fe:ea via ens192
Feb 24 15:49:20 foreman-new dhcpd: DHCPACK on (redacted).72 to 00:50:56:bc:fe:ea via ens192

Using a tftp client from the .72 VM, we can connect to the foreman server and “get pxelinux.0”.
[root@test-client ~]# tftp (redacted).48
tftp> get pxelinux.0
tftp> quit
[root@test-client ~]# ls -al pxelinux.0
-rw-r–r--. 1 root root 26746 Feb 24 16:06 pxelinux.0

I am perplexed why new hosts are working fine, but existing hosts are getting stuck. (It’s also very likely that I have messed something up during the foreman installation).

Thanks for looking!

#2

For anyone else experiencing this issue, it turned out to be a bad dhcp lease in “/var/lib/dhcpd/dhcpd.leases” which was overriding the next-server option with a “supersede server.next-server” line.
These are IP addresses presented in hexadecimal, and it was pointing to the wrong server somehow.
Manually editing the leases file with the correct IP (in hex) fixed the booting issue.

To prevent the problem appearing again, I had to edit “/etc/foreman-proxy/settings.d/tftp.yml” and un-comment “:tftp_servername: x.y.z.a” (giving it the IP of my foreman server).

#3

Yep, this happened to me as well. Note if you do this change, you also need to redeploy TFTP/DHCP configuration to all hosts which has been created before this configuration change. You can do this via mass action in Foreman.

1 Like