Image-based provisioned VMs won't boot off its disk file

Konstantin_Orekhov · March 21, 2015, 12:02am

Hello all!

I've got a strange problem going on with image-based provisioning into
Libvirt compute resources.

I have several QCOW2 images that have been verified to be bootable if used
as a whole ones.
However, when Foreman creates a new VM at my request, one of such images is
used as base image for that new VM disk (<my-vm-name>-disk1), but VM can
never boot off that disk for some reason.

If I stop a VM and manually copy over whatever a base image was used into
that <my-vm-name-disk1> file and start a VM after that, it has no issues
booting.

Has anyone seen such issues? If so, any advice?

I thought it could related to images but bringing CirrOS 0.3.3 yielded the
same results.

I'm running Foreman 1.7.1 (built with the installer) on RHEL 6.4. My KVM
HVs configured as Libvirt compute resources are ubuntu 12.04.2 and 14.04.1
ones. Both exhibit the same issues described above.

BTW, I noticed that all VMs created by my Foreman on Ubuntu 12.04 HVs have
this boot order for some reason:

while all VMs created on ubuntu14.04 HVs have expected boot order:

What's the reason for this? I can't think of one specifically for
image-based VMs, which are supposed to boot off the HD right away.

Thanks!

Konstantin_Orekhov · March 23, 2015, 5:37pm

Also, if someone has this available from top of their head - where do I
find the args Foreman uses to execute virt-install with during VM
(image-based) creation?

Konstantin_Orekhov · March 24, 2015, 12:59am

I tried to go through the code looking for "qemu" or "virt-install"
commands and can't find any of those.
Anyone is able to point me in proper direction?

···

On Monday, March 23, 2015 at 10:37:38 AM UTC-7, Konstantin Orekhov wrote: > > Also, if someone has this available from top of their head - where do I > find the args Foreman uses to execute virt-install with during VM > (image-based) creation? >

dLobatog · March 24, 2015, 7:53am

Hi Konstantin,

We rely on fog for these tasks:
https://github.com/fog/fog/tree/master/lib/fog/libvirt

The relevant code for creating a new Libvirt VM is mostly here:

github.com

theforeman/foreman/blob/develop/app/models/compute_resources/foreman/model/libvirt.rb#L98


  return unless new_volume_errors.empty?
  client.volumes.new(attr.merge(:allocation => '0G'))
end


def new_volume_errors
  errors = []
  errors.push _('no storage pool available on hypervisor') if storage_pools.empty?
  errors
end


def storage_pools
  client.pools rescue []
end


def bridges
  # before ruby-libvirt fixes https://bugzilla.redhat.com/show_bug.cgi?id=1317909 we have to use raw XML to get type
  bridges = client.client.list_all_interfaces.select do |libvirt_interface|
    type_match = libvirt_interface.xml_desc.match /<interface.*?type=['"]([a-z]+)['"]/
    type_match[1] == 'bridge'
  end
  bridge_names = bridges.map(&:name)

client.servers.new on new_vm is what gets us the fog object and we also
create the volumes on create_vm. Lines 111 and 112 set up the boot order
I think.

It sets up only <boot dev='hd'> unless you're provisioning from
network, in which case it also sets up 'network' first. If I'm reading
this correctly, you should only get <boot dev='network'> if you're not
doing image based provisioning on Foreman.

If you're clicking on image provisioning but you're getting this order
it looks like a bug to me. Let us know if you find more clues as to
what's happening and file a bug issue & ping us here or on IRC so
someone takes a shot at fixing it

Thanks !

···

On 03/23, 'Konstantin Orekhov' via Foreman users wrote: > I tried to go through the code looking for "qemu" or "virt-install" > commands and can't find any of those. > Anyone is able to point me in proper direction? > > On Monday, March 23, 2015 at 10:37:38 AM UTC-7, Konstantin Orekhov wrote: > > > > Also, if someone has this available from top of their head - where do I > > find the args Foreman uses to execute virt-install with during VM > > (image-based) creation? > > > > -- > You received this message because you are subscribed to the Google Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.

–
Daniel Lobato Garcia

@eLobatoss
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

Konstantin_Orekhov · March 25, 2015, 6:56pm

Hi, Daniel! Thanks for a reply, it is good to hear from you again!

>
> Hi Konstantin,
>
> We rely on fog for these tasks:
> https://github.com/fog/fog/tree/master/lib/fog/libvirt
> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Ffog%2Ffog%2Ftree%2Fmaster%2Flib%2Ffog%2Flibvirt&sa=D&sntz=1&usg=AFQjCNEK2XbRc20QYNwCESl8E1V-ysu7UA>
>

But how and what parameters Fog is being called by Foreman (during VM
creation)?

>
> The relevant code for creating a new Libvirt VM is mostly here:
>
> https://github.com/theforeman/foreman/blob/develop/app/models/compute_resources/foreman/model/libvirt.rb#L98
>
> client.servers.new on new_vm is what gets us the fog object and we also
> create the volumes on create_vm. Lines 111 and 112 set up the boot order
> I think.
>
> It sets up only <boot dev='hd'> unless you're provisioning from
> network, in which case it also sets up 'network' first. If I'm reading
> this correctly, you should only get <boot dev='network'> if you're not
> doing image based provisioning on Foreman.
>
> If you're clicking on image provisioning but you're getting this order
> it looks like a bug to me. Let us know if you find more clues as to
> what's happening and file a bug issue & ping us here or on IRC so
> someone takes a shot at fixing it
>
>
Yes, I think this is exactly what's happening. For whatever reason, it
seems that the "image_id" is not be being passed, so boot order gets
"network" inserted as the first device.
I'd also guess that missing image_id would be a culprit for a VM image
being created w/o a proper backing image defined thus making such VM
unbootable off that disk.
Is there a way for me to trace the calls after UI or API to see all the
data/params being passed around?

I'll submit an issue shortly. I replicate this very easily through both UI
and API calls where I specifically set image-based provisioning is to be
used.
Thanks!

···

On Tuesday, March 24, 2015 at 12:53:16 AM UTC-7, Daniel Lobato wrote:

ohadlevy · March 25, 2015, 7:03pm

> Hi, Daniel! Thanks for a reply, it is good to hear from you again!
>
>>
>> Hi Konstantin,
>>
>> We rely on fog for these tasks:
>> https://github.com/fog/fog/tree/master/lib/fog/libvirt
>> <https://www.google.com/url?q=https%3A%2F%2Fgithub.com%2Ffog%2Ffog%2Ftree%2Fmaster%2Flib%2Ffog%2Flibvirt&sa=D&sntz=1&usg=AFQjCNEK2XbRc20QYNwCESl8E1V-ysu7UA>
>>
>
> But how and what parameters Fog is being called by Foreman (during VM
> creation)?
>
>
>>
>> The relevant code for creating a new Libvirt VM is mostly here:
>> https://github.com/theforeman/foreman/blob/develop/app/
>> models/compute_resources/foreman/model/libvirt.rb#L98
>>
>> client.servers.new on new_vm is what gets us the fog object and we also
>> create the volumes on create_vm. Lines 111 and 112 set up the boot order
>> I think.
>>
>> It sets up only <boot dev='hd'> unless you're provisioning from
>> network, in which case it also sets up 'network' first. If I'm reading
>> this correctly, you should only get <boot dev='network'> if you're not
>> doing image based provisioning on Foreman.
>>
>> If you're clicking on image provisioning but you're getting this order
>> it looks like a bug to me. Let us know if you find more clues as to
>> what's happening and file a bug issue & ping us here or on IRC so
>> someone takes a shot at fixing it
>>
>>
> Yes, I think this is exactly what's happening. For whatever reason, it
> seems that the "image_id" is not be being passed, so boot order gets
> "network" inserted as the first device.
> I'd also guess that missing image_id would be a culprit for a VM image
> being created w/o a proper backing image defined thus making such VM
> unbootable off that disk.
> Is there a way for me to trace the calls after UI or API to see all the
> data/params being passed around?
>

sure, just look at the logs when you submit / POST, it should show all
parameters passed down.

Ohad

···

On Wed, Mar 25, 2015 at 8:56 PM, 'Konstantin Orekhov' via Foreman users < foreman-users@googlegroups.com> wrote: > On Tuesday, March 24, 2015 at 12:53:16 AM UTC-7, Daniel Lobato wrote:

I’ll submit an issue shortly. I replicate this very easily through both UI
and API calls where I specifically set image-based provisioning is to be
used.
Thanks!

–
You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Konstantin_Orekhov · March 25, 2015, 7:22pm

http://projects.theforeman.org/issues/9903

Taking yet another look at the logs - I couldn't find anything useful
before.

Konstantin_Orekhov · March 25, 2015, 10:08pm

False alarm, this is not a bug. My KVM HV had issues with apparmor
settings. After running these commands, image-based VM provisioning through
UI works as expected. API call needs to be reviewed and fixed as it
apparently does something else wrong.

$ sudo apt-get install apparmor-profiles apparmor-utils
$ sudo aa-complain /usr/lib/libvirt/virt-aa-helper