Problem: Intermittently, when provisioning a VM to a KVM we get this race condition, whereby the VM gets booted before the boot images are downloaded on the smart proxy. It’s like it’s not waiting for a handoff from the smart proxy.
The iPXE boot starts and gets the following during PXE boot on the console:
No more network devices
Booting from Hard Disk…
Boot failed: not a bootable disk
No bootable device.
Expected outcome:
I expect Foreman to wait for a handoff from the smart proxy, confirming the boot images like vmlinuz and init.rd are downloaded and ready before actually booting the VM.
Additional expectation, in the event of this type of boot failure, in general, the iPXE boot should wait for a bit and just automatically reboot. A manual reboot solves this problem 99% of the time.
Other relevant data:
Nothing notable in the debug logs.
I researched this a couple of weeks ago and swear I found a bug in history that was purportedly fixed for this problem. I can’t find this presently.
We know about this bug for years, unfortunately the solution is a little bit more complicated, we would like to move PXE files download from the moment when host is created to the moment when operation system is created/updated.
If a file is already present, we use “wget -c” (continue) flag which does not redownload the contents, so you are fine and once you hit this bug and file a bug report, it’s already “fixed” on your deployment There are other issues around “-c” we want to tackle in short term, but it is what it is for now.
Of course. The iPXE image that boots when a Foreman virtual machine is deployed, goes through the process to check for PXE server availability, and if it doesn’t find a server, it tries a local boot.
The local boot should obviously not work, so it will say “there were no boot devices found…”
Then it just sits there. It never reboots after a timeout or anything. Maybe that’s by design?
I feel like it should reboot automatically after X seconds. I’m not an image expert, though I would dare to hack the image to allow such a thing, if it’s possible.
If it did reboot, then we wouldn’t really care about this race condition, as a reboot fixes the problem 99% of the time.
Hey, it’s difficult to understand your workflow when you mix iPXE with PXE, these are two different things. I still don’t need understand - do you see “no devices found” printed by PXELinux or iPXE? Are you using Bootdisk (then which disk image) or just iPXE from VM firmware?
I think you are using bootdisk - host or generic image and when MAC address is not found in Foreman, you get an error. You can of course “hack” the disk very easily - there are two templates you can actually change, just search “bootdisk” in your templates, these are here:
When there is no match, it simply prints an error, you can change that to sleep and then reboot - whatever you like. Then regenerate your bootdisk.
Foreman asks us to choose both a PXELinux template, and an iPXE Template. There is nothing visible about a bootdisk, though I’m thinking that it could be something called from the PXELinux template.
Please see the following for the exact point of failure, it appears like it’s happening after a PXE boot attempt fails:
Unfortunately I can’t tell from the screenshot if you are PXE booting or doing iPXE boot, because the output is very similar. Let me guess, what you are doing here is PXE boot. We have no control on how virtual BIOS work after all boot methods are tried (network, hard drive). Usually, computer hangs and that’s what you see here, so it’s expected behavior.
It works, and after 5 seconds the system reboots if no media was found.
So our next mission is to figure out how to either get this as a default on all libvirt hosts, or, to figure out how Foreman calls libvirt remotely and sets up the XML / imports the VM - perhaps add a feature to auto-reboot, inject it in the import during provisioning?
To commit autoreboot as seen above, we decided to modify the XML directly, so, for example in our Puppet repo we edited site/profile/files/foreman/server.xml.erb such that it looks like the following (note the line that says “bios useserial”):
<domain type='<%= domain_type %>'>
<name><%= name %></name>
<memory><%= memory_size %></memory>
<vcpu><%= cpus %></vcpu>
<os>
<type arch='<%= arch %>'><%= os_type %></type>
<% boot_order.each do |dev| -%>
<boot dev='<%= dev %>'/>
<% end -%>
<bios useserial='yes' rebootTimeout='5000'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<% if !cpu.empty? -%>
<% if cpu[:mode] -%>
<cpu mode='<%= cpu[:mode] %>'>
<% else -%>
<cpu>
<% end -%>
<%
fallback = 'allow'
model = ''
if cpu[:model]
fallback = cpu[:model][:fallback] if cpu[:model][:fallback]
model = cpu[:model][:name] if cpu[:model][:name]
end
-%>
<model fallback='<%= fallback %>'><%= model %></model>
</cpu>
<% end -%>
<clock offset='utc'/>
<devices>
<% volumes.each do |vol| -%>
<disk type='file' device='disk'>
<driver name='qemu' type='<%= vol.format_type %>'/>
<source file='<%= vol.path %>'/>
<%# we need to ensure a unique target dev -%>
<target dev='vd<%= ('a'..'z').to_a[volumes.index(vol)] %>' bus='virtio'/>
</disk>
<% end -%>
<% if iso_file -%>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='<%= "#{iso_dir}/#{iso_file}" %>'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='1' unit='0'/>
</disk>
<% end -%>
<% nics.each do |nic| -%>
<interface type='<%= nic.type %>'>
<source <%= nic.type == 'bridge' ? "bridge='#{nic.bridge}'" : "network='#{nic.network}'" %> />
<model type='<%= nic.model %>'/>
</interface>
<% end -%>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target port='0'/>
</console>
<input type='tablet' bus='usb'/>
<input type='mouse' bus='ps2'/>
<%
display_type = display[:type]
if display[:port].empty?
display_port = display[:port]
autoport = 'no'
else
display_port = '-1'
autoport = 'yes'
end
unless display[:listen].empty?
display_listen = "listen='#{display[:listen]}'"
end
unless display[:password].empty?
display_password = "passwd='#{display[:password]}'"
end
-%>
<graphics type='<%= display_type %>' port='<%= display_port %>' autoport='<%= autoport %>' <%= display_listen %> <%= display_password %> />
<video>
<model type='cirrus' vram='9216' heads='1'/>
</video>
</devices>
</domain>
We also added the following to site/profile/manifests/foreman.pp
# A fix for the PXE issue. This will inject the tag <bios useserial='yes' rebootTimeout='5000'/>
file { '/opt/theforeman/tfm/root/usr/share/gems/gems/fog-libvirt-0.4.1/lib/fog/libvirt/models/compute/templates/server.xml.erb':
ensure => present,
owner => 'root',
group => 'root',
mode => '0644',
source => [ 'puppet:///modules/profile/foreman/server.xml.erb' ],
}
This will possibly fail with updates, so a better permanent solution possibly through Fog update is required.
There are other issues around “-c” we want to tackle in short term, but it is what it is for now.