Foreman 2.0 is not creating PXE boot files

ramuskay · June 4, 2020, 3:21pm

Problem:

Since I update from 1.24 to 2.0 Foreman is not creating PXE boot files, my provision host can grab a DHCP IP but then it can’t boot and i got this (see below) :

Expected outcome:

In 1.24 it was booting correctly.

Foreman and Proxy versions:

Name	Version
foreman	2.0
Ansible	3.0.1
DHCP	2.0.0
DNS	2.0.0
Dynflow	0.2.4
HTTPBoot	2.0.0
SSH	0.3.0
TFTP	2.0.0

Foreman and Proxy plugin versions:

Name	Version
foreman	2.0
foreman-tasks	1.1.1
foreman_ansible	5.0.1
foreman_remote_execution	3.2.1
foreman_setup	7.0.0

Distribution and version:

Description: CentOS Linux release 7.7.1908 (Core)

Other relevant data:

I noticed that the file in my tftp corresponding to my macadress was not created, here the logs from the /var/log/messages when i was trying to perform a provisionning :

> Jun  4 16:48:56 pl1-foreman in.tftpd[6265]: RRQ from 10.201.180.104 filename pxelinux.0
> Jun  4 16:48:56 pl1-foreman in.tftpd[6265]: Client 10.201.180.104 finished pxelinux.0
> Jun  4 16:48:56 pl1-foreman in.tftpd[6266]: RRQ from 10.201.180.104 filename pxelinux.cfg/4210d399-e807-58a0-2e8b-d5e028eb6c0a
> Jun  4 16:48:56 pl1-foreman in.tftpd[6266]: Client 10.201.180.104 File not found pxelinux.cfg/4210d399-e807-58a0-2e8b-d5e028eb6c0a
> Jun  4 16:48:56 pl1-foreman in.tftpd[6267]: RRQ from 10.201.180.104 filename pxelinux.cfg/01-00-50-56-90-9d-03
> Jun  4 16:48:56 pl1-foreman in.tftpd[6267]: Client 10.201.180.104 File not found pxelinux.cfg/01-00-50-56-90-9d-03
> Jun  4 16:48:56 pl1-foreman in.tftpd[6268]: RRQ from 10.201.180.104 filename pxelinux.cfg/0AC9B468
> Jun  4 16:48:56 pl1-foreman in.tftpd[6268]: Client 10.201.180.104 File not found pxelinux.cfg/0AC9B468

However in the provisionning wizard the PXELoader was correctly offer, it is correctly link to the Operating System and Hostgroup:

I browse the others topics related to my issue but none resolved it. The problem appears when I updated Foreman from 1.24 to 2.0

lzap · June 5, 2020, 7:25am

Hey,

Do you have PXELinux template associated with your OS?
Does PXELinux template render correctly?
Do you have TFTP proxy associated with the provisioning Subnet?
Do you pick the correct Subnet and OS?
Is TFTP directory writeable to the foreman-proxy user?
Are there any SELinux denials?
Have you investigated proxy.log when the host is being created? Any errors there?
And finally - enable debug mode for both foreman and foreman-proxy, restart those services and then pastebin whole transaction (grep request id) from both production.log and proxy.log (the same uuid).

ramuskay · June 5, 2020, 8:45am

Hey thank you @lzap for your answer.

Yes, in my Operating system :
I don’t really understand your question, but i can browse it in Foreman and it shows no error
Yes
Yes

And
Yes

[root@pl1-foreman tftpboot]# pwd
/var/lib/tftpboot
[root@pl1-foreman tftpboot]# ll
total 188
drwxr-xr-x. 2 foreman-proxy root 4096 Jun 3 11:37 boot
-rw-r–r–. 1 foreman-proxy root 20832 Apr 11 21:40 chain.c32
drwxr-xr-x. 2 foreman-proxy root 4096 Jun 5 09:45 grub
drwxr-xr-x. 2 foreman-proxy root 4096 Jun 5 09:45 grub2
-rw-r–r–. 1 foreman-proxy root 33628 Apr 11 21:40 mboot.c32
-rw-r–r–. 1 foreman-proxy root 26140 Apr 11 21:40 memdisk
-rw-r–r–. 1 foreman-proxy root 55140 Apr 11 21:40 menu.c32
drwxr-xr-x. 2 foreman-proxy root 6 Apr 11 21:40 poap.cfg
-rw-r–r–. 1 foreman-proxy root 26759 Apr 11 21:40 pxelinux.0
drwxr-xr-x. 2 foreman-proxy root 4096 Jun 5 09:45 pxelinux.cfg
drwxr-xr-x. 2 foreman-proxy root 6 Apr 11 21:40 ztp.cfg

SELinux is permissive
Yes, no error in there
FYI I replace real domain name by <domain_name>
/var/log/foreman-proxy/proxy.log : /var/log/foreman-proxy/proxy.log - Pastebin.com
/var/log/foreman/production.log : /var/log/foreman/production.log - Pastebin.com

lzap · June 5, 2020, 12:04pm

Now that’s weird. There is a check before TFTP is even attempted to be queued, can you put some debuging statements in there:

github.com

theforeman/foreman/blob/develop/app/models/concerns/orchestration/tftp.rb#L18


  before_destroy :queue_tftp_destroy


  # required for pxe template url helpers
  include Rails.application.routes.url_helpers
  register_rebuild(:rebuild_tftp, N_('TFTP'))
end


def tftp_ready?
  # host.managed? and managed? should always come first so that orchestration doesn't
  # even get tested for such objects
  (host.nil? || host&.managed?) && managed && provision? && (host&.operatingsystem && host.pxe_loader.present?) && !image_build? && SETTINGS[:unattended]
end


def tftp?
  tftp_ready? && !!(subnet && subnet.tftp?)
end


def tftp6?
  tftp_ready? && !!(subnet6 && subnet6.tftp?)
end

Try to print all those preconditions. You can simply use Rails.logger.info host&.managed? etc to see them all. Restart and dig this in the logs, something must be missing. We need to improve how we report these preconditions.

ramuskay · June 5, 2020, 12:52pm

Re @Izap i think i figured out, I seems that i can’t “manage” my two interfaces. When I try to tick the checkbox “Manage” in one of my interfaces it untick automatically the other. Is it normal ?

In 1.24 i didn’t have this issue, I edit one of my host I provisionned with 1.24 my two interfaces are “Manage” :

So I tried to manage the provision interface, the provision worked but my ansible playbook didn’t worked (actually maybe not related) :

TimoGoebel · June 5, 2020, 1:25pm

Is it this bug? https://github.com/theforeman/foreman/pull/7693

ramuskay · June 5, 2020, 1:28pm

@TimoGoebel Yes absolutely. So the fix was pushed in the develop branch ?

TimoGoebel · June 5, 2020, 1:29pm

According to Bug #29709: Only one interface can be marked as Managed - Foreman it should go into 2.0.1.

ramuskay · June 5, 2020, 1:30pm

Thank you @TimoGoebel and @lzap for your support

lzap · June 5, 2020, 1:42pm

Thanks Timo, I had no idea about this one.

These pre-orchestration checks needs better logging. It happens way too often. I will add them some day