Tftp failure on foreman 3.8.0

Support
,
1

Problem:
Tftp does not serve boot files, installer did not properly setup up tftp so I started systemctl service tftp
Expected outcome:
Foreman should create a proper tftp service or EL8.9 tftp for systemctl should have a doc with proper options

Foreman and Proxy versions:
Foreman version foreman-3.8.0-1.el8.noarch proxy version foreman-proxy-3.8.0-1.el8.noarch
Foreman and Proxy plugin versions:
unsure
Distribution and version:
Red Hat Enterprise Linux release 8.9 (Ootpa)
Other relevant data:

an old example of a Xinetd tftp file from our old broken system is here

service tftp
{
port = 69
disable = no
socket_type = dgram
protocol = udp
wait = yes
user = root
group = root
groups = yes
server = /usr/sbin/in.tftpd
server_args = -v -s /var/lib/tftpboot -m /etc/tftpd.map
per_source = 11
cps = 100 2
flags = IPv4
instances = UNLIMITED
}

I got this to start w xinetd in rhel8.9 on the new system but when I would connect via tftp client it just hung so I stopped it and then tried a similar config on systemctl
i.e.
I made sure tftp.socket was running 1st and then started it and reloaded etc

[root@pulp3 log]# cat /usr/lib/systemd/system/tftp.service
[Unit]
Description=Tftp Server
Requires=tftp.socket
Documentation=man:in.tftpd

[Service]

ExecStart=/usr/sbin/in.tftpd -c -4 -v -s /var/lib/tftpboot
StandardInput=socket

[Install]
Also=tftp.socket

I can use a client to download files its a little slow however

[root@pulp3 log]# systemctl status tftp.service
● tftp.service - Tftp Server
Loaded: loaded (/usr/lib/systemd/system/tftp.service; indirect; vendor preset: disabled)
Active: inactive (dead) since Tue 2024-08-13 16:19:53 PDT; 11min ago
Docs: man:in.tftpd
Process: 13017 ExecStart=/usr/sbin/in.tftpd -c -4 -v -s /var/lib/tftpboot (code=exited, status=0/SUCCESS)
Main PID: 13017 (code=exited, status=0/SUCCESS)

Aug 13 16:04:17 pulp3.gld.x.net in.tftpd[13067]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 13 16:04:21 pulp3.gld.x.net in.tftpd[13068]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 13 16:04:25 pulp3.gld.x.net in.tftpd[13070]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 13 16:04:33 pulp3.gld.x.net in.tftpd[13072]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 13 16:04:37 pulp3.gld.x.net in.tftpd[13073]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 13 16:04:41 pulp3.gld.x.net in.tftpd[13075]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 13 16:04:45 pulp3.gld.x.net in.tftpd[13076]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 13 16:04:49 pulp3.gld.x.net in.tftpd[13077]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 13 16:04:53 pulp3.gld.x.net in.tftpd[13079]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 13 16:19:53 pulp3.gld.x.net systemd[1]: tftp.service: Succeeded.

Problem is why I try and kick a node it times out and says as well

Aug 13 16:04:53 pulp3 systemd[1]: Received notify message exceeded maximum size. Ignoring.

I assume I need some minor way to correct that or a proper way to have foreman create the service , Suggustions? I’d really appreciate it.

2

TFTP is not setup by default anymore, so you need to specify the parameter of the installer to spin it up. If done so, I never had a problem with it yet.

Also do not try to setup it with xinetd, we have now socket based activation via systemd instead.

3

the fact that on rhel8.9 it no longer uses xinetd is great, however on our system it did not setup systemd for tftp correctly that is what I am trying to figure out please advise i.e. what does a systemd config file look like?

4

Note that we set up tftp.socket by default. This means the service is started as needed.

Can you share how you set up your system? Preferably the installer command including arguments (and please redact passwords if any).

5

thanks

here is the info again thx

foreman-installer --scenario katello --enable-foreman-proxy --foreman-proxy-tftp=true --foreman-proxy-tftp-listen-on=“https”

nothing was created for services it seems but it allowed kickstart files to be created in ‘/var/lib/tftpboot’ so I changed the config files to

[root@pulp3 ~]# cat /etc/foreman-proxy/settings.d/tftp.yml
---
# TFTP management
:enabled: true
:tftproot: /var/lib/tftpboot

then I enabled and turned on and enabled tftp.socket and tftp.service

[root@pulp3 ~]# systemctl status tftp.socket
● tftp.socket - Tftp Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/tftp.socket; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2024-08-13 14:14:14 PDT; 19h ago
   Listen: [::]:69 (Datagram)
    Tasks: 0 (limit: 1231977)
   Memory: 0B
   CGroup: /system.slice/tftp.socket

Aug 13 14:14:14 pulp3.gld.dreamworks.net systemd[1]: Listening on Tftp Server Activation Socket.
[root@pulp3 ~]#

tftp service

[root@pulp3 ~]# systemctl status  tftp.service
● tftp.service - Tftp Server
   Loaded: loaded (/usr/lib/systemd/system/tftp.service; indirect; vendor preset: disabled)
   Active: active (running) since Wed 2024-08-14 09:09:51 PDT; 21s ago
     Docs: man:in.tftpd
 Main PID: 52473 (in.tftpd)
    Tasks: 1 (limit: 1231977)
   Memory: 208.0K
   CGroup: /system.slice/tftp.service
           └─52473 /usr/sbin/in.tftpd -c -B 65464 -p -v -s /var/lib/tftpboot -u root --ipv4 --timeout 300

Aug 14 09:09:51 pulp3.gld.dreamworks.net systemd[1]: Started Tftp Server.
Aug 14 09:10:06 pulp3.gld.dreamworks.net in.tftpd[52481]: RRQ from ::ffff:100.110.69.147 filename pxelinux.0
Aug 14 09:10:06 pulp3.gld.dreamworks.net in.tftpd[52481]: Client ::ffff:100.110.69.147 finished pxelinux.0
[root@pulp3 ~]# systemctl cat  tftp.service
# /usr/lib/systemd/system/tftp.service
[Unit]
Description=Tftp Server
Requires=tftp.socket
Documentation=man:in.tftpd

[Service]
# ExecStart=/usr/sbin/in.tftpd -s /var/lib/tftpboot
# ExecStart=/usr/sbin/in.tftpd -c -v -s /var/lib/tftpboot  -u tftp --ipv4 --timeout 300
ExecStart=/usr/sbin/in.tftpd -c -B 65464 -p -v -s /var/lib/tftpboot  -u root --ipv4 --timeout 300
# ExecStart=/usr/sbin/in.tftpd -c -p -v -s /var/lib/tftpboot  -u root --ipv4 --timeout 300 -r tsize -r blksize
# -m /etc/tftpd.map
StandardInput=socket

[Install]
Also=tftp.socket

so if I try and manually fetch tftp it works

tftp> get pxelinux.0
getting from pulp3.x.net:pxelinux.0 to pxelinux.0 [netascii]
Received 42945 bytes in 0.0 seconds [20108779 bit/s]

if I start to kickstart a machine it shows it tries to download the nbp file and the logs show it

Aug 14 09:11:40 pulp3 in.tftpd[52502]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 14 09:11:41 pulp3 systemd[1]: Received notify message exceeded maximum size. Ignoring.
Aug 14 09:11:48 pulp3 systemd[1]: message repeated 7 times: [Received notify message exceeded maximum size. Ignoring.]
Aug 14 09:11:49 pulp3 in.tftpd[52503]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 14 09:11:49 pulp3 in.tftpd[52503]: tftp: client does not accept options
Aug 14 09:11:49 pulp3 in.tftpd[52505]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 14 09:11:49 pulp3 systemd[1]: Received notify message exceeded maximum size. Ignoring.
Aug 14 09:11:52 pulp3 systemd[1]: message repeated 3 times: [Received notify message exceeded maximum size. Ignoring.]
Aug 14 09:11:53 pulp3 in.tftpd[52507]: RRQ from ::ffff:100.110.70.131 filename grub2/grubx64.efi
Aug 14 09:11:53 pulp3 systemd[1]: Received notify message exceeded maximum size. Ignoring.

but then it times out, if I did something wrong I am fine with doing the right but how?

6

I want to add
selinux is disabled and the node rebooted its still disabled.
firewalld is off and ipatables flushed manaully
no nft tables & flushed manually in case

7

ranran installer with options

foreman-installer --scenario katello
–enable-foreman-proxy
–foreman-proxy-httpboot true
–foreman-proxy-http true
–foreman-proxy-tftp true
–foreman-proxy-tftp-listen-on=“https”

and still not able to provision I am sure this is per showing the request for filename grub2/grubx64.efi is httpboot and foreman appears to have turned on the tftp service now automatically so likely its still the options

8

kickstart files have a ref to the foreman/katello server with port 8000 but I see no service runing on port 8000 is that normal is it spawned?

9

You don’t need to specify listen-on. It’s an advanced parameter that you really shouldn’t touch unless you know what you’re doing.

The installer will overwrite this. Funnily enough you also explicitly reverted the value for --foreman-proxy-tftp-listen-on to both. This is less secure and not recommended. Nothing should need to access the Smart Proxy’s TFTP module over plain text HTTP.

The installer should do this already.

I see you’ve changed the TFTP service. I’d strongly recommend against changing /usr/lib/systemd/system/tftp.service and instead rely on drop in files. For example, using systemctl edit tftp. For example:

[Service]
ExecStart=
ExecStart=/usr/sbin/in.tftpd -c -B 65464 -p -v -s /var/lib/tftpboot  -u root --ipv4 --timeout 300

There actually is a PR that I’ve neglected for too long to make this easier to tune using the installer (Manage more parameters extension by dvo-rak · Pull Request #141 · theforeman/puppet-tftp · GitHub).

I’d also clarify the design because I think you’re misunderstanding the fundamental design.

Foreman needs to control the files on the TFTP server. Because those typically don’t have a REST API we have implemented this in Foreman Proxy. Its entire purpose is to manage files in the tftproot (/var/lib/tftproot). That’s it. If the files show up there, don’t touch it because it’s doing its job.

As for TFTP: it’s using UDP and known be unreliable and sensitive to both firewalls and latency. I see you already found httpboot and I hope you read Provisioning Hosts

You should see the smart-proxy process (from foreman-proxy.service) listening on port 8000 after you ran the installer. If it’s not, you need to check the logs because something is wrong.

10

ok uninstalled tftp-server then reinstalled it verified its config is default,

read over doc , regarding Grub2 UEFI HTTP option the target kickstart host complains about that possibly not allow a node to boot of it and does not pick it up thus fails.

I actually dont have a preference of whether we want uefi http or not we just want to be able to provision nodes and those nodes used the old method of grub2 uefi in our broken 1.8.0 system.

reran installer excluding bind to port https

restarted foreman proxy and refreshed features, attempted again and it still doesn’t work, tried using stock pxegrub2 template

11

also I can fetch

http://pulp3.x.x.net:8000/httpboot/grub2/grubx64.efi via web browser file so that shows its now got port 8000 I looked in settings.yml and it had a comment that it was disabled by default so enabled it

12

if I try and hit

http://pulp3.x.x.net/:8000/httpboot/pxelinux.cfg/default
I get redirected to https and an error message

The page you were looking for doesn’t exist.

You may have mistyped the address or the page may have moved.

If you are the application owner check the logs for more information.

13

if I try and look at the ks file as shown by the pxe file then I get ( not sure if this is normal or not)

Failed to proxy /provision for {“token”=>“4147f377-c674-460c-b508-f6b9b7bdd430”, “kind”=>“provision”}: Error retrieving unattended/provision for {“token”=>“4147f377-c674-460c-b508-f6b9b7bdd430”, “url”=>“http://pulp3.x.x.net:8000”} from pulp3.x.x.net: Net::HTTPInternalServerError: 500

14

perms on the boot dir are

[root@pulp3 lib]# ls -alrt /var/lib/| grep tftpboot
drwxr-xr-x. 9 root root 4096 Aug 14 11:54 tftpboot

15

ok so maybe this will help its not writing the ks file i.e. I should have

/var/lib/tftpboot/grub2/grub.cfg-01-e0-4f-43-e6-7e-b7 but I do not

what could cause this?

16

ok Im gonna make a wild guess and fill in on something that I didnt want to have to go through making this ticket long but its already getting long.

The old system Died some files were backed up as it was in the progress of being migrated pretty much I see that all services are running in the new system but one thing I do not see if puppet, you had pointed this out before but perhaps I assumed that this was always gonna be installed and just not shown in the ui. Could that be the issue in the foreman-proxy that it needs that specifically set in its configs and enabled with the installer?

Also tftpboot was populated with a backup from the dead server nothing else was copied over and files with the old server name like pxe files were removed.

this is why initially my question was specifiaclly about pxe as everything else is working i.e. I can fetch a dhcp ip from infoblox and creds can do stuff in ipa i.e. realm.

if puppet options are actually needed to manage tftp and that is what is going on that would explain it. if its bundled then something perhaps other is off.

17

ran

foreman-installer
–enable-foreman-cli-puppet
–enable-foreman-plugin-puppet
–enable-puppet
–foreman-proxy-puppet true
–foreman-proxy-puppetca true
–puppet-server true

and it enable puppetca in the smart proxy options still can’t kick a node yet

18

ok its writing ks files now at least

19

example of generated config

This file was deployed via ‘Custom Kickstart default PXEGrub2’ template

set default=0
set timeout=10

menuentry ‘Custom Kickstart default PXEGrub2’ {
linuxefi boot/centostestv2-UPd1pzG0DVwJ-vmlinuz ks=http://pulp3.x.x.net:8000/unattended/provision?token=8eb555c2-76c3-4a3b-81b9-022709537558 network ksdevice=bootif ks.device=bootif BOOTIF=00-e0-4f-43-e6-7e-b7 kssendmac ks.sendmac inst.ks.sendmac ip=dhcp
initrdefi boot/centostestv2-UPd1pzG0DVwJ-initrd.img
}

Smart proxy does not have HTTPBoot feature with HTTP port enabled, skipping EFI HTTP boot menu entry

menuentry ‘Custom Kickstart default PXEGrub2 EFI HTTPS’ --id efi_https {
linuxefi (https,100.110.x.x:9090)/httpboot/boot/centostestv2-UPd1pzG0DVwJ-vmlinuz ks=http://pulp3.x.x.net:8000/unattended/provision?token=8eb555c2-76c3-4a3b-81b9-022709537558 network ksdevice=bootif ks.device=bootif BOOTIF=01-e0-4f-43-e6-7e-b7 ks=http://pulp3.x.x.net:8000/unattended/provision?token=8eb555c2-76c3-4a3b-81b9-022709537558 kssendmac ks.sendmac ip=dhcp
initrdefi (https,100.110x.x:9090)/httpboot/boot/centostestv2-UPd1pzG0DVwJ-initrd.img
}

via curl response I get to http is

curl: (7) Failed to connect to pulp3.gld.x.net port 8000: Connection refused

and for 9090 If I try and curl to https I get

Failed to proxy /provision for {“token”=>“8eb555c2-76c3-4a3b-81b9-022709537558”, “kind”=>“provision”}: Error retrieving unattended/provision for {“token”=>“8eb555c2-76c3-4a3b-81b9-022709537558”, “url”=>“http://pulp3.gld.x.net:8000”} from pulp3.x.x.net: Net::HTTPInternalServerError: 500

20

so at current we have people who cannot work thus I was trying to keep this ticket simple , and ask for only what might be related to this one part of the process, since I currently cant get this working I am going to try and use an older version of foreman on a spare box that is CentOS7 to try and discern the difference I would prefer to use 3.80 so will be watching this ticket any help would be greatly appreciated.