TFTP boot images not found/regenerated

Support
1

Problem:
When booting a newly discovered host (discovery worked fine) I got this error:

filenotfound
filenotfound722×213 4.41 KB

I checked the /srv/tftp/boot/ directory: The file was there and had a size of about 30MB. I deleted the file, so it would be regenerated, when I click on “Build PXE Default” in Hosts > Provisioning Templates. But it did not get regenerated. Now, when I boot the new host I get: PXE-E32: TFTP Timeout.

Expected outcome:
The machine should boot into the OS installer.

Foreman and Proxy versions:
2.0.0

Foreman and Proxy plugin versions:
foreman-tasks 1.1.1
foreman_ansible 5.0.1
foreman_discovery 16.0.1
foreman_remote_execution 3.2.1

Distribution and version:
Debian 10.4

Other relevant data:
This (141.3 KB) is the output of foreman-installer --full-help
This (520 Bytes) is /var/log/foreman-proxy/proxy.log (I emptied it before trying the delete, so isolate the relevant logs)

2

Your networking is incorrect, check firewalls, note TFTP is UDP (stateless) protocol and it won’t work through NAT without special module.

Enable debug and restart foreman-proxy, you should see wget command with full URL and response. Investigate what was wrong.

Linux kernel should be 1MB is size, you probably saw initramdisk which is around 30-60MB depending on distro.

This all works out of box, you had to do something wrong.

3

Thank you for the quick reply!
There is no firewall in my lab network. Foreman (192.168.0.240/24) is in the same subnet as my host (192.168.0.9/24). The internet gateway (router) is 192.168.0.1. So, there is no NAT. But maybe I messed the configuration anyway.

How do I enable debug?

4

I enabled debug and restarted the foreman-proxy service. Then I click on “Build PXE Default” again. This is /var/log/foreman-proxy/proxy.log (2.0 KB)

5

I need to see log from Rails app: /var/log/foreman/production.log. Only the relevant transaction please.

6

I emptied the log before I clicked on the button. This is the log afterwards:
production.log (3.8 KB)

7

I did a complete clean new install on another server. The same problem is there too.

8

Does this TFTP boot/ image creating use the foreman_remote_execution plugin?

9

No, Foreman does HTTPS call to /tftp and the proxy downloads those files via wget. It’s that easy.

10

Share logs, those lines with wget. It’s usually easy to troubleshoot.

If files are there, then logs from system journal (TFTP service) is the starting point.

11

I searched all the logs I could find for wget. Then I found “wget” in /var/log/foreman-proxy/proxy.log:

2020-05-15T11:56:54 229f1c45 [D] Starting task: /usr/bin/wget --connect-timeout=10 --dns-timeout=10 --read-timeout=60 --tries=3 --no-check-certificate -nv -c “http://ftp.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux” -O “/srv/tftp/boot/debian-mirror-eZTy0v5ew0jk-linux”
2020-05-15T11:56:54 229f1c45 [D] Starting task: /usr/bin/wget --connect-timeout=10 --dns-timeout=10 --read-timeout=60 --tries=3 --no-check-certificate -nv -c “http://ftp.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz” -O “/srv/tftp/boot/debian-mirror-eZTy0v5ew0jk-initrd.gz”
2020-05-15T11:56:57 2e04c71c [D] Starting task: /usr/bin/wget --connect-timeout=10 --dns-timeout=10 --read-timeout=60 --tries=3 --no-check-certificate -nv -c “http://ftp.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/debian-installer/amd64/linux” -O “/srv/tftp/boot/debian-mirror-eZTy0v5ew0jk-linux”
2020-05-15T11:56:57 2e04c71c [D] Starting task: /usr/bin/wget --connect-timeout=10 --dns-timeout=10 --read-timeout=60 --tries=3 --no-check-certificate -nv -c “http://ftp.debian.org/debian/dists/buster/main/installer-amd64/current/images/netboot/debian-installer/amd64/initrd.gz” -O “/srv/tftp/boot/debian-mirror-eZTy0v5ew0jk-initrd.gz”

This was about an hour ago. Seems that there is some cron job or similar, which did that (I was afk the last hour). Now there are two more files in /srv/tftp/boot/:

debian-mirror-eZTy0v5ew0jk-initrd.gz
debian-mirror-eZTy0v5ew0jk-linux

I will try to provision a host as soon as I made progress with this: Creating discovered hosts is not working

To be honest, I am kind of surprised regarding the development of this topic. I had a similar experience before (Tasks is no longer working after update - #8 by kiigass). On one hand it is nice, that foreman is kind of “self-repairing” on the other hand I would have liked to know what the issue is and how to solve it, if it comes back.

This is the whole /var/log/foreman-proxy/proxy.log (70.5 KB)

12

Its not a cron job, wget are subprocesses of foreman-proxy and if the connection is slow, they can be downloading those files for minutes.

The best thing you can do is check md5 sum of these with versions you download via your browser.

Now, all the rest is not in Foreman’s hands. This is between TFTP client and TFTP server to do the work. Check firewalls etc.

13

The update of ruby-smart-proxy-remote-execution-ssh to version 0.3.0 fixed the issue with the creation of hosts and now the tftp boot is also working again. I dont really know if it was the update or the delayed download. I will mark your answer as solution. I really appreciate all your help, thank you!