Foreman isn't placing TFTP files into /var/lib/tftpboot/pxelinux.cfg


Host type: VMware VMs with vSphere
OS: Ubuntu 18.04 LTS

When I hit the ‘Build’ button, Foreman is not adding a file to the /var/lib/tftpboot/pxelinux.cfg directory. Therefore, when the host boots, it picks up the /var/lib/tftpboot/pxelinux.cfg/default and the OS is never installed.

After configuring a host, I can view the template from the ‘Templates > PXELinux template > View’ button.

When I boot a host, I can see the DHCP & TFTP traffic in the system logs. Notice how the request filename pxelinux.cfg/01-00-50-56-96-e8-f2 fails.

Oct  3 16:35:43 foreman dhcpd[1363]: DHCPACK on to 00:50:56:96:e8:f2 via ens224
Oct  3 16:35:43 foreman in.tftpd[3669]: RRQ from filename pxelinux.0
Oct  3 16:35:43 foreman in.tftpd[3669]: tftp: client does not accept options
Oct  3 16:35:43 foreman in.tftpd[3670]: RRQ from filename pxelinux.0
Oct  3 16:35:43 foreman in.tftpd[3671]: RRQ from filename ldlinux.c32
Oct  3 16:35:43 foreman in.tftpd[3672]: RRQ from filename pxelinux.cfg/42169ef8-AAAA-BBBB-CCCC-DDDDDDDD
Oct  3 16:35:43 foreman in.tftpd[3673]: RRQ from filename pxelinux.cfg/01-00-50-56-96-e8-f2
Oct  3 16:35:43 foreman in.tftpd[3674]: RRQ from filename pxelinux.cfg/D1A898ED
Oct  3 16:35:43 foreman in.tftpd[3675]: RRQ from filename pxelinux.cfg/D1A898E
Oct  3 16:35:43 foreman in.tftpd[3676]: RRQ from filename pxelinux.cfg/D1A898
Oct  3 16:35:43 foreman in.tftpd[3677]: RRQ from filename pxelinux.cfg/D1A89
Oct  3 16:35:43 foreman in.tftpd[3678]: RRQ from filename pxelinux.cfg/D1A8
Oct  3 16:35:43 foreman in.tftpd[3679]: RRQ from filename pxelinux.cfg/D1A
Oct  3 16:35:43 foreman in.tftpd[3680]: RRQ from filename pxelinux.cfg/D1
Oct  3 16:35:43 foreman in.tftpd[3681]: RRQ from filename pxelinux.cfg/D
Oct  3 16:35:43 foreman in.tftpd[3682]: RRQ from filename pxelinux.cfg/default
Oct  3 16:35:43 foreman in.tftpd[3683]: RRQ from filename menu.c32
Oct  3 16:35:43 foreman in.tftpd[3683]: tftp: client does not accept options
Oct  3 16:35:43 foreman in.tftpd[3684]: RRQ from filename menu.c32
Oct  3 16:35:43 foreman in.tftpd[3685]: RRQ from filename libutil.c32
Oct  3 16:35:43 foreman in.tftpd[3686]: RRQ from filename pxelinux.cfg/default
  • The operating system Ubuntu 18.04 has a PXELinux template associated with it:

Likewise, the Hostgroup is set to use ‘PXE loader’ of type ‘PXELinux BIOS’:

Expected outcome:

When I hit the ‘Build’ button, I expected Foreman to create a file like /var/lib/tftpboot/pxelinux.cfg/01-00-11-22-33-AA-BB-CC which will then be picked up when the host is booting.

Foreman and Proxy versions:

Foreman Server: 1.23.0
Smart Proxies: Logs, TFTP, DHCP, Puppet CA, Puppet, and HTTPBoot 1.23.0

Foreman and Proxy plugin versions:

Foreman Discovery 15.1.0

Other relevant data:

Can you try without hostgroup? HG inheritance can contain bugs.

Also the host or hostgroup must not have Image-based provision flag set.

Can you explain what you mean by this lzap? I’m looking inside my hostgroup and I don’t see any place to check where an Image-based provision flag could be set.

When I check which templates that are applied, I see the following:

This happens with or without the hostgroup.

Stefan, are you sure that the Subnet associated with the host has TFTP Smart Proxy associated? No TFTP orchestration is happening unless it’s there.

Yes, the Subnet definitely has a TFTP Proxy associated with it.

The TFTP proxy is now populating /var/lib/tftpboot/pxelinux.cfg after I disabled the Cloud-init template.

So this brings up the question: Can I use the same Operating System for both a PXEboot-style provisioning and Cloudinit/image-based provisioning? It seems that some of the Operating System parameters are exclusive to one or the other.

whats in /var/log/foreman-proxy/proxy.log // /var/log/foreman/production.log ?

foreman-proxy + tftp-server are able to write to /var/lib/tftpboot?

Ooh, yeah. We do have a check - in case of image-based provisioning TFTP is completely skipped:

(host.nil? || host&.managed?) && managed && provision? && (host&.operatingsystem && host.pxe_loader.present?) && !image_build? && SETTINGS[:unattended]

Operating system is not an issue, you can share it. The problem is that Provision Method was set to image. Can you confirm?

I’d be very surprised if this was caused simply by associating cloud-init template. Very surprised. :wink:

No, the Provision Method is set to “Network”. This is a VM on VMware, but we’re using network provisioning in this particular case.

One of the host groups that I used doesn’t actually have a default network associated with it. This is because our facility has a dozen different networks which would normally be selected when the system if provisioned-- one of these happens to be the provisioning network, and we select it at provisioning time when necessary.

Is this related?

production.log has dozens and dozens of lines, most of which look related to the UI rendering. It’s hard to find important things in there.

proxy.log says this after I hit the ‘Build’ button":

2019-10-23T09:54:41 9a6e173c [I] Started DELETE /puppet/ca/
2019-10-23T09:54:42 9a6e173c [E] Attempt to remove nonexistent client certificate for
2019-10-23T09:54:42 9a6e173c [I] Finished DELETE /puppet/ca/ with 404 (1503.07 ms)
2019-10-23T09:54:42 9a6e173c [I] Started POST /puppet/ca/autosign/
2019-10-23T09:54:42 9a6e173c [I] Finished POST /puppet/ca/autosign/ with 200 (0.71 ms)

Yes and yes. Keep in mind that the Foreman Proxy writes files to this directory, which are later read by the tftp-server. I don’t believe that tftp-server actually writes files here.

Well if you don’t select a subnet, then any orchestration can happen. But it’s hard to tell from what I’ve seen.

the error doesn’t seem to happen on the proxy, the /foreman/production.log would be interesting, specially what happens when you trigger the error.

btw. you are aware that foreman and foreman-proxy uses different users with different groups? so if you enabled the user foreman to write to /var/lib/tftpboot its wrong

@lzap i think he uses subnet, he has it just not configured to the hostgroup because he picks the subnet when he provision a vm as i understand.

This is correct.

Enable debug mode in Foreman (Rails), restart and do it again, then pastebin only the transaction that is relevant. While it might not make sense to you, there is valuable info on orchestration. Some conditions must be met to have TFTP action scheduled.