ERF42-3305 configuring networkless template-based provisionning on libvirt/KVM/LVM

Jqueuniet · November 9, 2015, 9:36am

Greetings,

I'm trying to setup Foreman, got it working with LAN hypervisors, but now
for production purposes, I need to get it up running for remote VM install
on rented servers where I have close to no control over the network. We get
no access to DHCP, VM have to be configured statically with a somewhat
unconventionnal network setup, a /32 hostmask using the hypervisor as a
manual route and default gateway. We used to install those VM networkless
and setup the network during post-install via VNC. Since standard
network-based installs doesn't seem to cut it in those conditions, I wanted
to try out templates in this environnement.

So I configured manually my template VM, installed cloud-init, made sure it
ran, wrote some custom cloud-init logic in Foreman to setup the network
correctly but I can't seem to get past the second step in host creation,
after userdata rendering and during resources allocations. The process
stops with the following error message :

Failed to create a compute HV (Libvirt) instance vm.fqdn: ERF42-3305
[Foreman::Exception]: Unable to find template /dev/vm_kvm/tplvm

The overall setup is using libvirt with KVM and a LVM storage pool. I
RTFMed a bit, mostly found messages related to the template image path
( ERF42-3305 - Foreman ). I'm
not too sure how I fall into this problem since the image path seems to
correspond with how the storage pool was configured, but I'm probably just
missing something.

Here is my pool definition :

virsh pool-dumpxml vm_kvm

Here is my template definition :

virsh dumpxml tplvm

<domain type='kvm'>
<name>tplvm</name>
<uuid>50234253-c419-6f3d-7003-0dcbf8063eac</uuid>
<description>xxxxxxxxxxxxxxxxxxxxxxx</description>
<memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory>
<vcpu placement='static'>2</vcpu>
<os>
<type arch='x86_64' machine='pc-1.1'>hvm</type>
<boot dev='hd'/>
</os>
<features>
<acpi/>
<apic/>
<pae/>
</features>
<clock offset='utc'/>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='block' device='disk'>
<driver name='qemu' type='raw' cache='none'/>
<source dev='/dev/vm_kvm/tplvm'/>
<target dev='vda' bus='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04'
function='0x0'/>
</disk>
<disk type='block' device='cdrom'>
<driver name='qemu' type='raw'/>
<target dev='hdc' bus='ide'/>
<readonly/>
<address type='drive' controller='0' bus='1' target='0' unit='0'/>
</disk>
<controller type='usb' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x2'/>
</controller>
<controller type='ide' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x01'
function='0x1'/>
</controller>
<interface type='bridge'>
<mac address='xx:xx:xx:xx:xx:xx'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03'
function='0x0'/>
</interface>
<serial type='pty'>
<target port='0'/>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<input type='mouse' bus='ps2'/>
<graphics type='vnc' port='5915' autoport='no' listen='0.0.0.0'
keymap='fr'>
<listen type='address' address='0.0.0.0'/>
</graphics>
<video>
<model type='cirrus' vram='9216' heads='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02'
function='0x0'/>
</video>
<memballoon model='virtio'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05'
function='0x0'/>
</memballoon>
</devices>
</domain>

I should probably add I'm using Foreman 1.9.3. The Foreman host and the VM
are running Debian 8, the hypervisor is still running Debian 7.The HV is
running on bare-metal Dell and HP servers rented with Online (
https://www.online.net/ ).

Thanks in advance.

Gwmngilfen · November 10, 2015, 5:07pm

The short answer, from me anyway, is that I have no idea how that
error is coming about. What I can do is tell you how I'm doing my
image-based libvirt stuff, and see if we can compare notes

So I use a single qcow image for my base VM - it lives in
/home/baseimages on my hypervisor. I've added an "Image" to the
hypervisor in the Foreman UI using that path. So, it doesn't have any
XML in the libvirt hypervisor itself, it's just a disk file. It does
use cloud-init, using NoCloud local files ofc, and grow-rootfs.

Then when I create a host in Foreman, the XML looks like [1] - the
interesting bit being:

&lt;disk type=&#39;file&#39; device=&#39;disk&#39;&gt;
  &lt;driver name=&#39;qemu&#39; type=&#39;qcow2&#39;/&gt;
  &lt;source file=&#39;/var/lib/libvirt/images/test.primary.sapphire.local-disk1&#39;/&gt;
  &lt;backingStore type=&#39;file&#39; index=&#39;1&#39;&gt;
    &lt;format type=&#39;qcow2&#39;/&gt;
    &lt;source file=&#39;/home/baseimages/debian-wheezy-7.0.0-3-amd64.qcow2&#39;/&gt;
    &lt;backingStore/&gt;
  &lt;/backingStore&gt;
  &lt;target dev=&#39;vda&#39; bus=&#39;virtio&#39;/&gt;
  &lt;alias name=&#39;virtio-disk0&#39;/&gt;
  &lt;address type=&#39;pci&#39; domain=&#39;0x0000&#39; bus=&#39;0x00&#39; slot=&#39;0x04&#39;

function='0x0'/>
</disk>

So, the interesting bit for, and the only things I can see as being
different, is (a) that you're using LVM, but I think that should work,
and (b) a whole template vm rather than just a backing store. Have you
tried just using the disk from your base vm as a backing store?

Greg

[1] http://pastebin.com/hXpvdg8V

Jqueuniet · November 12, 2015, 9:38am

Thanks for your answer Greg.

I actually already tried something like that and duplicated the LVM disk
inside a raw image file with dd. I declared the storage directory as a
file-based pool in libvirt, with the resulting XML definition:

<pool type='dir'>
<name>templates</name>
<uuid>5a4d3f4b-0753-0861-5a7f-a03ecc08bbd5</uuid>
<capacity unit='bytes'>787444981760</capacity>
<allocation unit='bytes'>27824562176</allocation>
<available unit='bytes'>759620419584</available>
<source>
</source>
<target>
<path>/opt/img</path>
<permissions>
<mode>0700</mode>
<owner>4294967295</owner>
<group>4294967295</group>
</permissions>
</target>
</pool>

And the failure on VM creation is even weirder, I get a low-level error
with a failed LVM command where Foreman tries to do a LVM snapshot with the
raw image file as source, and fails since AFAIK snapshots only work in an
LVM-to-LVM scenario.

Failed to create a compute HV (Libvirt) instance vm.fqdn: Call to
virNetworkCreateXML failed: internal error Child process (/sbin/lvcreate
–name vm.fqdn-disk1 -L 20971520K -s /opt/img/tplvm.raw) unexpected exit
status 3: "/opt/img/tplvm.raw": Invalid path for Logical Volume The origin
name should include the volume group. Run `lvcreate --help' for more
information.

I'm not sure how to interpret that, if I have to stay on LVM the whole way
if the destination is on LVM. And I can't really test an
imagefile-to-imagefile scenario since the destination has to be LVM. The
formating and content seem to imply this is a libvirt errror rather than a
Foreman error.

Sadly, I can't compare with your XML file since I never get to the point
where the VM is actually created. Maybe I'll try to dig this backing store
idea, I never saw any mention about that up to now. Is it some kind of
prebaked debian image disk where you just didn't do a full-fledged install?

Felix_Barbeira · June 16, 2016, 4:12pm

Have you checked linux permissions for that directory? I had the same error
and fix it giving "libvirt-qemu:kvm" owners to directory and base image.

···

El lunes, 9 de noviembre de 2015, 10:36:44 (UTC+1), jque...@amabis.fr escribió: > > Greetings, > > I'm trying to setup Foreman, got it working with LAN hypervisors, but now > for production purposes, I need to get it up running for remote VM install > on rented servers where I have close to no control over the network. We get > no access to DHCP, VM have to be configured statically with a somewhat > unconventionnal network setup, a /32 hostmask using the hypervisor as a > manual route and default gateway. We used to install those VM networkless > and setup the network during post-install via VNC. Since standard > network-based installs doesn't seem to cut it in those conditions, I wanted > to try out templates in this environnement. > > So I configured manually my template VM, installed cloud-init, made sure > it ran, wrote some custom cloud-init logic in Foreman to setup the network > correctly but I can't seem to get past the second step in host creation, > after userdata rendering and during resources allocations. The process > stops with the following error message : > > Failed to create a compute HV (Libvirt) instance vm.fqdn: ERF42-3305 > [Foreman::Exception]: Unable to find template /dev/vm_kvm/tplvm > > The overall setup is using libvirt with KVM and a LVM storage pool. I > RTFMed a bit, mostly found messages related to the template image path > ( http://projects.theforeman.org/projects/foreman/wiki/ERF42-3305 ). I'm > not too sure how I fall into this problem since the image path seems to > correspond with how the storage pool was configured, but I'm probably just > missing something. > > Here is my pool definition : > > # virsh pool-dumpxml vm_kvm > > vm_kvm > 82ee3e78-94bd-053f-c982-dca2f9b09ec9 > 2193688100864 > 1911260446720 > 282427654144 > > vm_kvm > > > > /dev/vm_kvm > > 0700 > 4294967295 > 4294967295 > > > > > Here is my template definition : > > # virsh dumpxml tplvm > > tplvm > 50234253-c419-6f3d-7003-0dcbf8063eac > xxxxxxxxxxxxxxxxxxxxxxx > 2097152 > 2097152 > 2 > > hvm > > > > > > > > > destroy > restart > restart > > /usr/bin/kvm > > > > > function='0x0'/> > > > > > > > > > function='0x2'/> > > > function='0x1'/> > > > > > > function='0x0'/> > > > > > > > > > keymap='fr'> > > > > > function='0x0'/> > > > function='0x0'/> > > > > > I should probably add I'm using Foreman 1.9.3. The Foreman host and the VM > are running Debian 8, the hypervisor is still running Debian 7.The HV is > running on bare-metal Dell and HP servers rented with Online ( > https://www.online.net/ ). > > Thanks in advance. >

Gwmngilfen · November 12, 2015, 10:05am

> I'm not sure how to interpret that, if I have to stay on LVM the whole way
> if the destination is on LVM. And I can't really test an
> imagefile-to-imagefile scenario since the destination has to be LVM. The
> formating and content seem to imply this is a libvirt errror rather than a
> Foreman error.

Yeah, that looks like a bundle of fun It does seem to be expected
an LVM device. I wonder if you could deboostrap into a mounted LVM LV
and use it as a backing store (see below)… That might work.

> Sadly, I can't compare with your XML file since I never get to the point
> where the VM is actually created. Maybe I'll try to dig this backing store
> idea, I never saw any mention about that up to now. Is it some kind of
> prebaked debian image disk where you just didn't do a full-fledged install?

Kinda both. It's not a full-install in the sense of creating a VM -
instead I use a script which is a tweaked version of the official
Debian disk builder tool[1] which will basically loopback-mount a
file, run debootstrap into it, add some customization, and then
convert it to a qcow2 file. This leaves me with a fully installed
disk, rather than a complete VM definition in libvirt. It may be
possible to do this to an LVM LV instead of a loopbacked file, with
some effort.

I did a little digging, and the Libvirt docs seem to suggest you can
use a device rather than a file for a backingstore:

&lt;disk type=&#39;file&#39; device=&#39;disk&#39;&gt;
  &lt;driver name=&#39;qemu&#39; type=&#39;qcow2&#39;/&gt;
  &lt;source file=&#39;/var/lib/libvirt/images/domain.qcow&#39;/&gt;
  &lt;backingStore type=&#39;file&#39;&gt;
    &lt;format type=&#39;qcow2&#39;/&gt;
    &lt;source file=&#39;/var/lib/libvirt/images/snapshot.qcow&#39;/&gt;
    &lt;backingStore type=&#39;block&#39;&gt;
      &lt;format type=&#39;raw&#39;/&gt;
      &lt;source dev=&#39;/dev/mapper/base&#39;/&gt;
      &lt;backingStore/&gt;
    &lt;/backingStore&gt;
  &lt;/backingStore&gt;
  &lt;target dev=&#39;vdd&#39; bus=&#39;virtio&#39;/&gt;
&lt;/disk&gt;

Here we see a 3-level chain, "domain.qcow" which is based on
"snapshot.qcow" which is based on "/dev/mapper/base". That tells me
that you should be able to use an LVM as a backing store - try doing
that instead of a full VM?

Greg

[1] https://packages.debian.org/jessie/openstack-debian-images

···

On 12 November 2015 at 09:38, wrote:

Felix_Barbeira · June 18, 2016, 1:06pm

No, it is failing again. It looks like it is not related with linux
permissions. I post my configuration, so maybe anybody could help me. This
is my storage pool:

virsh # pool-info baseimages
Name: baseimages
UUID: ec526631-8997-4be8-9a18-d60d12634bda
State: running
Persistent: yes
Autostart: yes
Capacity: 98.31 GiB
Allocation: 1.53 GiB
Available: 96.78 GiB

virsh #

virsh # pool-dumpxml baseimages
<pool type='fs'>
<name>baseimages</name>
<uuid>ec526631-8997-4be8-9a18-d60d12634bda</uuid>
<capacity unit='bytes'>105555197952</capacity>
<allocation unit='bytes'>1638027264</allocation>
<available unit='bytes'>103917170688</available>
<source>
<device path='/dev/vg0/libvirt-storage'/>
<format type='auto'/>
</source>
<target>
<path>/home/baseimages</path>
<permissions>
<mode>0777</mode>
<owner>0</owner>
<group>0</group>
</permissions>
</target>
</pool>

virsh #

I already tried to delete the pool, create it again and even refresh. But
when I try to provision a new host, foreman fails with the following
message:

2016-06-18T14:41:22 [app] [I] Started POST "/hosts" for 192.168.0.156 at
2016-06-18 14:41:22 +0200
2016-06-18T14:41:22 [app] [I] Processing by HostsController#create as /
2016-06-18T14:41:22 [app] [I] Parameters: {"utf8"=>"✓",
"authenticity_token"=>"xxxxxxxxxxxWLzUzeo=", "host"=>{"name"=>"test1",
"hostgroup_id"=>"1", "compute_resource_id"=>"2", "environment_id"=>"",
"managed"=>"true", "progress_report_id"=>"[FILTERED]",
"type"=>"Host::Managed", "interfaces_attributes"=>{"0"=>{"_destroy"=>"0",
"type"=>"Nic::Managed", "mac"=>"", "identifier"=>"", "name"=>"test1",
"domain_id"=>"1", "subnet_id"=>"1", "ip"=>"192.168.0.45", "managed"=>"1",
"primary"=>"1", "provision"=>"1", "virtual"=>"0", "tag"=>"",
"attached_to"=>"", "compute_attributes"=>{"type"=>"bridge",
"bridge"=>"br0", "model"=>"virtio"}}}, "compute_attributes"=>{"cpus"=>"1",
"memory"=>"805306368", "start"=>"1",
"volumes_attributes"=>{"0"=>{"_delete"=>"", "pool_name"=>"baseimages",
"capacity"=>"10G", "allocation"=>"0G", "format_type"=>"qcow2"}},
"image_id"=>"/home/baseimages/ubuntu16/xenial-server-cloudimg-amd64-disk1.img"},
"architecture_id"=>"1", "operatingsystem_id"=>"1",
"provision_method"=>"image", "build"=>"1", "medium_id"=>"7",
"ptable_id"=>"62", "disk"=>"", "root_pass"=>"[FILTERED]",
"is_owned_by"=>"3-Users", "enabled"=>"0", "comment"=>"",
"overwrite"=>"false"}, "capabilities"=>"build image",
"provider"=>"Libvirt", "bare_metal_capabilities"=>"build"}
2016-06-18T14:41:22 [app] [W] Failed to create a compute kvm03 (Libvirt)
instance test1.domain.lan: ERF42-3305 [Foreman::Exception]: Unable to find
template /home/baseimages/ubuntu16/xenial-server-cloudimg-amd64-disk1.img
>
> Foreman::Exception: ERF42-3305 [Foreman::Exception]: Unable to find
template /home/baseimages/ubuntu16/xenial-server-cloudimg-amd64-disk1.img
>
/usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:99:in
template' > /usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:204:increate_volumes'
>
/usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:126:in
create_vm' > /usr/share/foreman/app/models/concerns/orchestration/compute.rb:76:insetCompute'
> /usr/share/foreman/app/models/concerns/orchestration.rb:162:in execute' > /usr/share/foreman/app/models/concerns/orchestration.rb:107:inblock in
process'
> /usr/share/foreman/app/models/concerns/orchestration.rb:99:in each' > /usr/share/foreman/app/models/concerns/orchestration.rb:99:inprocess'
> /usr/share/foreman/app/models/concerns/orchestration.rb:35:in on_save' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:429:inblock in make_lambda'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:161:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:161:inblock in halting'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:501:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:501:inblock in call'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:501:in
each' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:501:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:86:in
run_callbacks' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/callbacks.rb:302:increate_or_update'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/persistence.rb:103:in
save' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/validations.rb:51:insave'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/attribute_methods/dirty.rb:21:in
save' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:268:inblock (2 levels) in save'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:329:in
block in with_transaction_returning_status' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/connection_adapters/abstract/database_statements.rb:201:inblock in transaction'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in
within_new_transaction' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/connection_adapters/abstract/database_statements.rb:201:intransaction'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:208:in
transaction' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:326:inwith_transaction_returning_status'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:268:in
block in save' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:283:inrollback_active_record_state!'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:267:in
save' > /usr/share/foreman/app/models/concerns/foreman/sti.rb:29:insave_with_type'
> /usr/share/foreman/app/controllers/hosts_controller.rb:91:in create' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/implicit_render.rb:4:insend_action'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/abstract_controller/base.rb:189:in
process_action' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/rendering.rb:10:inprocess_action'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/abstract_controller/callbacks.rb:20:in
block in process_action' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:113:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:113:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:552:inblock (2 levels) in compile'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:495:in
block (2 levels) in around' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in
block (2 levels) in halting' > /usr/share/foreman/vendor/ruby/2.1.0/gems/rails-observers-0.1.2/lib/rails/observers/action_controller/caching/sweeping.rb:73:inaround'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:452:in
public_send' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:452:inblock in make_lambda'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:inblock in halting'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:inblock in around'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:495:in
block (2 levels) in around' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in
block (2 levels) in halting' > /usr/share/foreman/app/controllers/concerns/application_shared.rb:13:inset_timezone'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:429:in
block in make_lambda' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in
block in halting' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in
block in around' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:495:inblock (2 levels) in around'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:inblock (2 levels) in halting'
> /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in
clear_thread' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:429:inblock in make_lambda'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:inblock in halting'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:inblock in around'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:495:in
block (2 levels) in around' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in
block (2 levels) in halting' > /usr/share/foreman/vendor/ruby/2.1.0/gems/rails-observers-0.1.2/lib/rails/observers/action_controller/caching/sweeping.rb:73:inaround'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:452:in
public_send' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:452:inblock in make_lambda'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:inblock in halting'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:inblock in around'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:86:in
run_callbacks' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/abstract_controller/callbacks.rb:19:inprocess_action'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/rescue.rb:29:in
process_action' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/instrumentation.rb:32:inblock in process_action'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/notifications.rb:159:in
block in instrument' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/notifications/instrumenter.rb:20:ininstrument'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/notifications.rb:159:in
instrument' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/instrumentation.rb:30:inprocess_action'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/params_wrapper.rb:250:in
process_action' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/railties/controller_runtime.rb:18:inprocess_action'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/abstract_controller/base.rb:136:in
process' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionview-4.1.14.2/lib/action_view/rendering.rb:30:inprocess'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal.rb:196:in
dispatch' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/rack_delegation.rb:13:indispatch'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal.rb:232:in
block in action' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/routing/route_set.rb:80:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/routing/route_set.rb:80:in
dispatch' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/routing/route_set.rb:48:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/journey/router.rb:73:in
block in call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/journey/router.rb:59:ineach'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/journey/router.rb:59:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/routing/route_set.rb:690:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/apipie-rails-0.3.6/lib/apipie/static_dispatcher.rb:65:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/apipie-rails-0.3.6/lib/apipie/extractor/recorder.rb:132:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/apipie-rails-0.3.6/lib/apipie/middleware/checksum_in_headers.rb:27:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/rack-1.5.5/lib/rack/etag.rb:23:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/rack-1.5.5/lib/rack/conditionalget.rb:35:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/rack-1.5.5/lib/rack/head.rb:11:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/middleware/params_parser.rb:27:in
call' > /usr/share/foreman/lib/middleware/catch_json_parse_errors.rb:9:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/middleware/flash.rb:254:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/rack-1.5.5/lib/rack/session/abstract/id.rb:225:incontext'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/rack-1.5.5/lib/rack/session/abstract/id.rb:220:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/middleware/cookies.rb:562:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/query_cache.rb:36:in
call' > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:621:incall'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/middleware/callbacks.rb:29:in
`block in call'
>
/usr/share/foreman/vendor/ruby/2.1.0/gems/a

Felix_Barbeira · June 18, 2016, 3:19pm

I answer myself. It looks like the base image must be in the root
directory of the storage pool, otherwise foreman could not detect it. Maybe
it does not support folders in the storage pool? After type this it worked:

$
mv /home/baseimages/ubuntu16/xenial-server-cloudimg-amd64-disk1.img /home/baseimages/xenial-server-cloudimg-amd64-disk1.img

Maybe a bug?

···

El sábado, 18 de junio de 2016, 15:06:04 (UTC+2), Félix Barbeira escribió: > > No, it is failing again. It looks like it is not related with linux > permissions. I post my configuration, so maybe anybody could help me. This > is my storage pool: > > virsh # pool-info baseimages > Name: baseimages > UUID: ec526631-8997-4be8-9a18-d60d12634bda > State: running > Persistent: yes > Autostart: yes > Capacity: 98.31 GiB > Allocation: 1.53 GiB > Available: 96.78 GiB > > virsh # > > virsh # pool-dumpxml baseimages > > baseimages > ec526631-8997-4be8-9a18-d60d12634bda > 105555197952 > 1638027264 > 103917170688 > > > > > > /home/baseimages > > 0777 > 0 > 0 > > > > > virsh # > > I already tried to delete the pool, create it again and even refresh. But > when I try to provision a new host, foreman fails with the following > message: > > 2016-06-18T14:41:22 [app] [I] Started POST "/hosts" for 192.168.0.156 at > 2016-06-18 14:41:22 +0200 > 2016-06-18T14:41:22 [app] [I] Processing by HostsController#create as */* > 2016-06-18T14:41:22 [app] [I] Parameters: {"utf8"=>"✓", > "authenticity_token"=>"xxxxxxxxxxxWLzUzeo=", "host"=>{"name"=>"test1", > "hostgroup_id"=>"1", "compute_resource_id"=>"2", "environment_id"=>"", > "managed"=>"true", "progress_report_id"=>"[FILTERED]", > "type"=>"Host::Managed", "interfaces_attributes"=>{"0"=>{"_destroy"=>"0", > "type"=>"Nic::Managed", "mac"=>"", "identifier"=>"", "name"=>"test1", > "domain_id"=>"1", "subnet_id"=>"1", "ip"=>"192.168.0.45", "managed"=>"1", > "primary"=>"1", "provision"=>"1", "virtual"=>"0", "tag"=>"", > "attached_to"=>"", "compute_attributes"=>{"type"=>"bridge", > "bridge"=>"br0", "model"=>"virtio"}}}, "compute_attributes"=>{"cpus"=>"1", > "memory"=>"805306368", "start"=>"1", > "volumes_attributes"=>{"0"=>{"_delete"=>"", "pool_name"=>"baseimages", > "capacity"=>"10G", "allocation"=>"0G", "format_type"=>"qcow2"}}, > "image_id"=>"/home/baseimages/ubuntu16/xenial-server-cloudimg-amd64-disk1.img"}, > "architecture_id"=>"1", "operatingsystem_id"=>"1", > "provision_method"=>"image", "build"=>"1", "medium_id"=>"7", > "ptable_id"=>"62", "disk"=>"", "root_pass"=>"[FILTERED]", > "is_owned_by"=>"3-Users", "enabled"=>"0", "comment"=>"", > "overwrite"=>"false"}, "capabilities"=>"build image", > "provider"=>"Libvirt", "bare_metal_capabilities"=>"build"} > 2016-06-18T14:41:22 [app] [W] Failed to create a compute kvm03 (Libvirt) > instance test1.domain.lan: ERF42-3305 [Foreman::Exception]: Unable to find > template /home/baseimages/ubuntu16/xenial-server-cloudimg-amd64-disk1.img > > > > Foreman::Exception: ERF42-3305 [Foreman::Exception]: Unable to find > template /home/baseimages/ubuntu16/xenial-server-cloudimg-amd64-disk1.img > > > /usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:99:in > `template' > > > /usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:204:in > `create_volumes' > > > /usr/share/foreman/app/models/compute_resources/foreman/model/libvirt.rb:126:in > `create_vm' > > /usr/share/foreman/app/models/concerns/orchestration/compute.rb:76:in > `setCompute' > > /usr/share/foreman/app/models/concerns/orchestration.rb:162:in `execute' > > /usr/share/foreman/app/models/concerns/orchestration.rb:107:in `block > in process' > > /usr/share/foreman/app/models/concerns/orchestration.rb:99:in `each' > > /usr/share/foreman/app/models/concerns/orchestration.rb:99:in `process' > > /usr/share/foreman/app/models/concerns/orchestration.rb:35:in `on_save' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:429:in > `block in make_lambda' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:161:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:161:in > `block in halting' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:501:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:501:in > `block in call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:501:in > `each' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:501:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:86:in > `run_callbacks' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/callbacks.rb:302:in > `create_or_update' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/persistence.rb:103:in > `save' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/validations.rb:51:in > `save' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/attribute_methods/dirty.rb:21:in > `save' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:268:in > `block (2 levels) in save' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:329:in > `block in with_transaction_returning_status' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in > `block in transaction' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/connection_adapters/abstract/database_statements.rb:209:in > `within_new_transaction' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/connection_adapters/abstract/database_statements.rb:201:in > `transaction' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:208:in > `transaction' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:326:in > `with_transaction_returning_status' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:268:in > `block in save' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:283:in > `rollback_active_record_state!' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/transactions.rb:267:in > `save' > > /usr/share/foreman/app/models/concerns/foreman/sti.rb:29:in > `save_with_type' > > /usr/share/foreman/app/controllers/hosts_controller.rb:91:in `create' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/implicit_render.rb:4:in > `send_action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/abstract_controller/base.rb:189:in > `process_action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/rendering.rb:10:in > `process_action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/abstract_controller/callbacks.rb:20:in > `block in process_action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:113:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:113:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:552:in > `block (2 levels) in compile' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:495:in > `block (2 levels) in around' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in > `block (2 levels) in halting' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/rails-observers-0.1.2/lib/rails/observers/action_controller/caching/sweeping.rb:73:in > `around' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:452:in > `public_send' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:452:in > `block in make_lambda' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in > `block in halting' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in > `block in around' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:495:in > `block (2 levels) in around' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in > `block (2 levels) in halting' > > /usr/share/foreman/app/controllers/concerns/application_shared.rb:13:in > `set_timezone' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:429:in > `block in make_lambda' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in > `block in halting' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in > `block in around' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:495:in > `block (2 levels) in around' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in > `block (2 levels) in halting' > > /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in > `clear_thread' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:429:in > `block in make_lambda' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in > `block in halting' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in > `block in around' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:495:in > `block (2 levels) in around' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:306:in > `block (2 levels) in halting' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/rails-observers-0.1.2/lib/rails/observers/action_controller/caching/sweeping.rb:73:in > `around' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:452:in > `public_send' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:452:in > `block in make_lambda' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:305:in > `block in halting' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:494:in > `block in around' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:502:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/callbacks.rb:86:in > `run_callbacks' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/abstract_controller/callbacks.rb:19:in > `process_action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/rescue.rb:29:in > `process_action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/instrumentation.rb:32:in > `block in process_action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/notifications.rb:159:in > `block in instrument' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/notifications/instrumenter.rb:20:in > `instrument' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activesupport-4.1.14.2/lib/active_support/notifications.rb:159:in > `instrument' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/instrumentation.rb:30:in > `process_action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/params_wrapper.rb:250:in > `process_action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/activerecord-4.1.14.2/lib/active_record/railties/controller_runtime.rb:18:in > `process_action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/abstract_controller/base.rb:136:in > `process' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionview-4.1.14.2/lib/action_view/rendering.rb:30:in > `process' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal.rb:196:in > `dispatch' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal/rack_delegation.rb:13:in > `dispatch' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_controller/metal.rb:232:in > `block in action' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/routing/route_set.rb:80:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/routing/route_set.rb:80:in > `dispatch' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/routing/route_set.rb:48:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/journey/router.rb:73:in > `block in call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/journey/router.rb:59:in > `each' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/journey/router.rb:59:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/actionpack-4.1.14.2/lib/action_dispatch/routing/route_set.rb:690:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/apipie-rails-0.3.6/lib/apipie/static_dispatcher.rb:65:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/apipie-rails-0.3.6/lib/apipie/extractor/recorder.rb:132:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/apipie-rails-0.3.6/lib/apipie/middleware/checksum_in_headers.rb:27:in > `call' > > > /usr/share/foreman/vendor/ruby/2.1.0/gems/rack-1.5.5/lib/rack/etag.rb:23:in > `call' > > > /usr/share/foreman/vend