Foreman isn't placing TFTP files into /var/lib/tftpboot/pxelinux.cfg

Stefan_Lasiewski · October 4, 2019, 12:04am

Problem:

Host type: VMware VMs with vSphere
OS: Ubuntu 18.04 LTS

When I hit the ‘Build’ button, Foreman is not adding a file to the /var/lib/tftpboot/pxelinux.cfg directory. Therefore, when the host boots, it picks up the /var/lib/tftpboot/pxelinux.cfg/default and the OS is never installed.

After configuring a host, I can view the template from the ‘Templates > PXELinux template > View’ button.

When I boot a host, I can see the DHCP & TFTP traffic in the system logs. Notice how the request filename pxelinux.cfg/01-00-50-56-96-e8-f2 fails.

Oct  3 16:35:43 foreman dhcpd[1363]: DHCPACK on 192.168.100.100 to 00:50:56:96:e8:f2 via ens224
Oct  3 16:35:43 foreman in.tftpd[3669]: RRQ from 192.168.100.100 filename pxelinux.0
Oct  3 16:35:43 foreman in.tftpd[3669]: tftp: client does not accept options
Oct  3 16:35:43 foreman in.tftpd[3670]: RRQ from 192.168.100.100 filename pxelinux.0
Oct  3 16:35:43 foreman in.tftpd[3671]: RRQ from 192.168.100.100 filename ldlinux.c32
Oct  3 16:35:43 foreman in.tftpd[3672]: RRQ from 192.168.100.100 filename pxelinux.cfg/42169ef8-AAAA-BBBB-CCCC-DDDDDDDD
Oct  3 16:35:43 foreman in.tftpd[3673]: RRQ from 192.168.100.100 filename pxelinux.cfg/01-00-50-56-96-e8-f2
Oct  3 16:35:43 foreman in.tftpd[3674]: RRQ from 192.168.100.100 filename pxelinux.cfg/D1A898ED
Oct  3 16:35:43 foreman in.tftpd[3675]: RRQ from 192.168.100.100 filename pxelinux.cfg/D1A898E
Oct  3 16:35:43 foreman in.tftpd[3676]: RRQ from 192.168.100.100 filename pxelinux.cfg/D1A898
Oct  3 16:35:43 foreman in.tftpd[3677]: RRQ from 192.168.100.100 filename pxelinux.cfg/D1A89
Oct  3 16:35:43 foreman in.tftpd[3678]: RRQ from 192.168.100.100 filename pxelinux.cfg/D1A8
Oct  3 16:35:43 foreman in.tftpd[3679]: RRQ from 192.168.100.100 filename pxelinux.cfg/D1A
Oct  3 16:35:43 foreman in.tftpd[3680]: RRQ from 192.168.100.100 filename pxelinux.cfg/D1
Oct  3 16:35:43 foreman in.tftpd[3681]: RRQ from 192.168.100.100 filename pxelinux.cfg/D
Oct  3 16:35:43 foreman in.tftpd[3682]: RRQ from 192.168.100.100 filename pxelinux.cfg/default
Oct  3 16:35:43 foreman in.tftpd[3683]: RRQ from 192.168.100.100 filename menu.c32
Oct  3 16:35:43 foreman in.tftpd[3683]: tftp: client does not accept options
Oct  3 16:35:43 foreman in.tftpd[3684]: RRQ from 192.168.100.100 filename menu.c32
Oct  3 16:35:43 foreman in.tftpd[3685]: RRQ from 192.168.100.100 filename libutil.c32
Oct  3 16:35:43 foreman in.tftpd[3686]: RRQ from 192.168.100.100 filename pxelinux.cfg/default

The operating system Ubuntu 18.04 has a PXELinux template associated with it:

Likewise, the Hostgroup is set to use ‘PXE loader’ of type ‘PXELinux BIOS’:

Expected outcome:

When I hit the ‘Build’ button, I expected Foreman to create a file like /var/lib/tftpboot/pxelinux.cfg/01-00-11-22-33-AA-BB-CC which will then be picked up when the host is booting.

Foreman and Proxy versions:

Foreman Server: 1.23.0
Smart Proxies: Logs, TFTP, DHCP, Puppet CA, Puppet, and HTTPBoot 1.23.0

Foreman and Proxy plugin versions:

Foreman Discovery 15.1.0

Other relevant data:

lzap · October 7, 2019, 2:25pm

Can you try without hostgroup? HG inheritance can contain bugs.

Also the host or hostgroup must not have Image-based provision flag set.

Stefan_Lasiewski · October 19, 2019, 12:12am

Can you explain what you mean by this lzap? I’m looking inside my hostgroup and I don’t see any place to check where an Image-based provision flag could be set.

When I check which templates that are applied, I see the following:

Stefan_Lasiewski · October 19, 2019, 12:12am

This happens with or without the hostgroup.

lzap · October 21, 2019, 6:22am

Stefan, are you sure that the Subnet associated with the host has TFTP Smart Proxy associated? No TFTP orchestration is happening unless it’s there.

Stefan_Lasiewski · October 22, 2019, 11:22pm

Yes, the Subnet definitely has a TFTP Proxy associated with it.

Stefan_Lasiewski · October 22, 2019, 11:33pm

The TFTP proxy is now populating /var/lib/tftpboot/pxelinux.cfg after I disabled the Cloud-init template.

So this brings up the question: Can I use the same Operating System for both a PXEboot-style provisioning and Cloudinit/image-based provisioning? It seems that some of the Operating System parameters are exclusive to one or the other.

elytscha · October 23, 2019, 6:35am

whats in /var/log/foreman-proxy/proxy.log // /var/log/foreman/production.log ?

foreman-proxy + tftp-server are able to write to /var/lib/tftpboot?

lzap · October 23, 2019, 7:20am

Ooh, yeah. We do have a check - in case of image-based provisioning TFTP is completely skipped:

(host.nil? || host&.managed?) && managed && provision? && (host&.operatingsystem && host.pxe_loader.present?) && !image_build? && SETTINGS[:unattended]

Operating system is not an issue, you can share it. The problem is that Provision Method was set to image. Can you confirm?

I’d be very surprised if this was caused simply by associating cloud-init template. Very surprised.

Stefan_Lasiewski · October 23, 2019, 4:09pm

No, the Provision Method is set to “Network”. This is a VM on VMware, but we’re using network provisioning in this particular case.

Stefan_Lasiewski · October 23, 2019, 4:20pm

One of the host groups that I used doesn’t actually have a default network associated with it. This is because our facility has a dozen different networks which would normally be selected when the system if provisioned-- one of these happens to be the provisioning network, and we select it at provisioning time when necessary.

Is this related?

Stefan_Lasiewski · October 23, 2019, 4:59pm

production.log has dozens and dozens of lines, most of which look related to the UI rendering. It’s hard to find important things in there.

proxy.log says this after I hit the ‘Build’ button":

2019-10-23T09:54:41 9a6e173c [I] Started DELETE /puppet/ca/lora-keezer.example.org
2019-10-23T09:54:42 9a6e173c [E] Attempt to remove nonexistent client certificate for lora-keezer.example.org
2019-10-23T09:54:42 9a6e173c [I] Finished DELETE /puppet/ca/lora-keezer.example.org with 404 (1503.07 ms)
2019-10-23T09:54:42 9a6e173c [I] Started POST /puppet/ca/autosign/lora-keezer.example.org
2019-10-23T09:54:42 9a6e173c [I] Finished POST /puppet/ca/autosign/lora-keezer.example.org with 200 (0.71 ms)

Yes and yes. Keep in mind that the Foreman Proxy writes files to this directory, which are later read by the tftp-server. I don’t believe that tftp-server actually writes files here.

lzap · October 24, 2019, 6:51am

Well if you don’t select a subnet, then any orchestration can happen. But it’s hard to tell from what I’ve seen.

elytscha · October 24, 2019, 7:21am

the error doesn’t seem to happen on the proxy, the /foreman/production.log would be interesting, specially what happens when you trigger the error.

btw. you are aware that foreman and foreman-proxy uses different users with different groups? so if you enabled the user foreman to write to /var/lib/tftpboot its wrong

@lzap i think he uses subnet, he has it just not configured to the hostgroup because he picks the subnet when he provision a vm as i understand.

Stefan_Lasiewski · October 30, 2019, 6:39pm

This is correct.

lzap · November 19, 2019, 8:57am

Enable debug mode in Foreman (Rails), restart and do it again, then pastebin only the transaction that is relevant. While it might not make sense to you, there is valuable info on orchestration. Some conditions must be met to have TFTP action scheduled.