Virt lib and cached objects

Corey_Osman · December 23, 2011, 11:21pm

Ohad,

I added a caching mechanism to virt that allows a virt to greatly speedup the lookup process for libvirt and guest objects. The host object also automatically maintains the cache should any of those objects be removed outside the virt object.

However, after using with foreman I noticed that the new caching mechanism wasn't being utilized. I poked around the foreman code and noticed that foreman doesn't keep the virt host object in memory as it disconnects after making a call. This is why it takes a long time to re-index the vms with every refresh. Foreman has to recreate the virt host object along with all the guest objects and libvirt domain objects with every refresh or creation of a vm. Is there any way to keep the virt connection object in memory long enough for the virt caching mechanism to do its job? Is there any reason why we wouldn't want to keep the virt object in memory? The amount of time saved from keeping the virt object in memory would be about 20x improvement.

Is there any kind of top level foreman cache that can be used to store objects?

I put the virt cache in the host object should you want to take a look.

github.com

logicminds/virt/blob/master/lib/virt/host.rb

module Virt
  class Host < Virt::Connection
    # Represents a Physical host which runs the libvirt daemon

    attr_reader :connection
    def initialize
      @connection = Virt.connection.connection
    end

    def name
      connection.hostname
    end

    def guests
      {:running => running_guests, :defined => defined_guests}
    end

    def running_guests
      connection.list_domains.map do |domain|
        find_guest_by_id(domain)

This file has been truncated. show original

Corey Osman
corey@logicminds.biz

Green I.T and Datacenter Automation Specialist

ohadlevy · December 25, 2011, 7:12am

> Ohad,
>
> I added a caching mechanism to virt that allows a virt to greatly speedup
> the lookup process for libvirt and guest objects. The host object also
> automatically maintains the cache should any of those objects be removed
> outside the virt object.
>
awesome!

>
> However, after using with foreman I noticed that the new caching mechanism
> wasn't being utilized. I poked around the foreman code and noticed that
> foreman doesn't keep the virt host object in memory as it disconnects after
> making a call. This is why it takes a long time to re-index the vms with
> every refresh. Foreman has to recreate the virt host object along with all
> the guest objects and libvirt domain objects with every refresh or creation
> of a vm. Is there any way to keep the virt connection object in memory
> long enough for the virt caching mechanism to do its job? Is there any
> reason why we wouldn't want to keep the virt object in memory? The amount
> of time saved from keeping the virt object in memory would be about 20x
> improvement.
>

Yes, you can use Rails.cache, see how we cache puppet certificate calls
–>

github.com

ohadlevy/foreman/blob/0.4-stable/app/models/smart_proxies/puppet_ca.rb#L19


  @name, @state, @fingerprint, @valid_from, @expires_at, @smart_proxy_id = opts.flatten
  @valid_from = Time.parse(@valid_from) unless @valid_from.blank?
  @expires_at = Time.parse(@expires_at) unless @expires_at.blank?
end


  class << self


    def all(proxy)
      raise "Must specify a Smart Proxy to use" if proxy.nil?


      unless certs = Rails.cache.read("ca_#{proxy.id}")
        api = ProxyAPI::Puppetca.new({:url => proxy.url})


        certs = api.all.map do |name, properties|
          new([name.strip, properties['state'], properties['fingerprint'], properties["not_before"], properties["not_after"], proxy.id])
        end.compact


        # save our CA details for 5 seconds
        Rails.cache.write("ca_#{proxy.id}", certs, {:expires_in  => 1.minute })
      end
      certs

note also for line 27.

Ohad

···

On Sat, Dec 24, 2011 at 1:21 AM, Corey Osman wrote:

Is there any kind of top level foreman cache that can be used to store
objects?

I put the virt cache in the host object should you want to take a look.
https://github.com/logicminds/virt/blob/master/lib/virt/host.rb

Corey Osman
corey@logicminds.biz

Green I.T and Datacenter Automation Specialist