On Compute Resource capacity visibility

In exploring Foreman as a viable frontend for a large deployment of
libvirt-based hypervisors, the number one complaint I've received from
fellow engineers is how difficult it is to "see" the resource consumption
of libvirt hosts. That is, if I want to create a new VM, I want to choose a
host for the guest that isn't under heavy load, has disk space to spare,
enough memory for the new VM, and so on.

The solution I've relied on thus far has been to use virt-manager as a
centralized view for how heavily loaded individual hypervisors are, but
this isn't ideal. Centralizing all management through Foreman would be a
better solution (for most use cases - I understand that for other use
cases, other interfaces for libvirt are better suited for this.)

To that end, I'm plowing through Rails tutorials and the
foreman_plugin_template source code to create an at-a-glance view of
compute resource resource consumption and availability. It's slow going,
but the community is tremendously helpful and I've got enough direction to
figure it out from here (thanks for the help, Dominic!)

With that in mind, a few questions for the Foreman devs:

  • Is this the right way to tackle the problem, or does Foreman handle
    this in a way I haven't found yet? I know Foreman will withhold memory
    options above the threshhold that the hypervisor can allocate, but I've
    been bit by throwing VMs onto hypervisors without enough disk space (and
    similar experiences) that it warrants some sort of solution like what I'm
    talking about.
  • Are there any problems to this approach? The most apparent I've
    thought of is latency: if I want to see all hypervisors at-a-glance and am
    asking each in turn about their current memory, load, and disk space,
    things would get slow (I'm playing with Rails caching to solve this, but
    that's still seems like a lot of data to pull regardless)
  • Last, if this seems like a viable endeavor, are there examples of
    extending the API? I've got enough to go on to create some new dashboards
    and models, but I haven't seen a clear example demonstrating how to extend
    the API yet.

Thank you all for such a great open source solution, the fact that I can
whip up a plugin for a need like this so easily is incredible. If there's
any interest I can share the github repo as well for feedback.

Hi,

See answers below.

> In exploring Foreman as a viable frontend for a large deployment of
> libvirt-based hypervisors, the number one complaint I've received from
> fellow engineers is how difficult it is to "see" the resource consumption of
> libvirt hosts. That is, if I want to create a new VM, I want to choose a
> host for the guest that isn't under heavy load, has disk space to spare,
> enough memory for the new VM, and so on.
>
> The solution I've relied on thus far has been to use virt-manager as a
> centralized view for how heavily loaded individual hypervisors are, but this
> isn't ideal. Centralizing all management through Foreman would be a better
> solution (for most use cases - I understand that for other use cases, other
> interfaces for libvirt are better suited for this.)

I think, that once your libvirt infrastructure gets a bit too big, its
time to consider other alternatives (such as ovirt [1] ), foreman
would support them happily, and they would probably the better
solution down the road, as virtualization management is a large task
by itself.

once I've said that, I'll happy respond to your questions with regard
to implement it within foreman :slight_smile:
>
> To that end, I'm plowing through Rails tutorials and the
> foreman_plugin_template source code to create an at-a-glance view of compute
> resource resource consumption and availability. It's slow going, but the
> community is tremendously helpful and I've got enough direction to figure it
> out from here (thanks for the help, Dominic!)
>
> With that in mind, a few questions for the Foreman devs:
>
> Is this the right way to tackle the problem, or does Foreman handle this in
> a way I haven't found yet? I know Foreman will withhold memory options above
> the threshhold that the hypervisor can allocate, but I've been bit by
> throwing VMs onto hypervisors without enough disk space (and similar
> experiences) that it warrants some sort of solution like what I'm talking
> about.

All communications to virt/cloud infrastructure in foreman, happens
via the compute resource objects, which in turn, uses the fog [2]
library to do the actual communication to every provider.

this gives a lot of benefits, but, when a feature is missing, it
usually means you need to learn and ruby project (fog) in order to
implement the feature in foreman.

> Are there any problems to this approach? The most apparent I've thought of
> is latency: if I want to see all hypervisors at-a-glance and am asking each
> in turn about their current memory, load, and disk space, things would get
> slow (I'm playing with Rails caching to solve this, but that's still seems
> like a lot of data to pull regardless)

Yes, thats true, currently Fog will not help you much here, you would
need to query each and every hypervisor, an alternative (outside the
scope of fog and libvirt) could be to write an mcollective plugin,
which gets a list of requirements (xx cpus, yy memory, zz disks (each
with disk type and space etc) and respond with some sort of value, in
which, you could make a decision on which one is avail atm.

> Last, if this seems like a viable endeavor, are there examples of extending
> the API? I've got enough to go on to create some new dashboards and models,
> but I haven't seen a clear example demonstrating how to extend the API yet.

its pretty easy to extend the api via a plugin, its actually identical
to adding another controller, but instead of inheriting from the
application_controller, you do it from the api controller (v1 or v2 as
you prefer).

>
> Thank you all for such a great open source solution, the fact that I can
> whip up a plugin for a need like this so easily is incredible. If there's
> any interest I can share the github repo as well for feedback.

Please do share, we welcome any foreman plugin to join the foreman
github organization.

Hope this helps,
Ohad

[1] http://www.ovirt.org/
[2] http://fog.io/

··· On Sun, Jan 26, 2014 at 8:03 AM, wrote:

Great, thank you Ohad. Here's the plugin I'm working on:

https://github.com/qualtrics/foreman_hyperview

I'll be sure to keep oVirt in mind moving forward.

In the interim, are there some examples regarding how to package
foreman_plugins for inclusion in an rpm repository? I'm a bit of a new when
it comes to packaging gems into rpms. I assume there's probably a specfile
I can look at somewhere?

··· On Sunday, January 26, 2014 5:22:58 AM UTC-7, ohadlevy wrote: > > Hi, > > See answers below. > > On Sun, Jan 26, 2014 at 8:03 AM, <tyl...@qualtrics.com > > wrote: > > In exploring Foreman as a viable frontend for a large deployment of > > libvirt-based hypervisors, the number one complaint I've received from > > fellow engineers is how difficult it is to "see" the resource > consumption of > > libvirt hosts. That is, if I want to create a new VM, I want to choose a > > host for the guest that isn't under heavy load, has disk space to spare, > > enough memory for the new VM, and so on. > > > > The solution I've relied on thus far has been to use virt-manager as a > > centralized view for how heavily loaded individual hypervisors are, but > this > > isn't ideal. Centralizing all management through Foreman would be a > better > > solution (for most use cases - I understand that for other use cases, > other > > interfaces for libvirt are better suited for this.) > > I think, that once your libvirt infrastructure gets a bit too big, its > time to consider other alternatives (such as ovirt [1] ), foreman > would support them happily, and they would probably the better > solution down the road, as virtualization management is a large task > by itself. > > once I've said that, I'll happy respond to your questions with regard > to implement it within foreman :) > > > > To that end, I'm plowing through Rails tutorials and the > > foreman_plugin_template source code to create an at-a-glance view of > compute > > resource resource consumption and availability. It's slow going, but the > > community is tremendously helpful and I've got enough direction to > figure it > > out from here (thanks for the help, Dominic!) > > > > With that in mind, a few questions for the Foreman devs: > > > > Is this the right way to tackle the problem, or does Foreman handle this > in > > a way I haven't found yet? I know Foreman will withhold memory options > above > > the threshhold that the hypervisor can allocate, but I've been bit by > > throwing VMs onto hypervisors without enough disk space (and similar > > experiences) that it warrants some sort of solution like what I'm > talking > > about. > > All communications to virt/cloud infrastructure in foreman, happens > via the compute resource objects, which in turn, uses the fog [2] > library to do the actual communication to every provider. > > this gives a lot of benefits, but, when a feature is missing, it > usually means you need to learn and ruby project (fog) in order to > implement the feature in foreman. > > > Are there any problems to this approach? The most apparent I've thought > of > > is latency: if I want to see all hypervisors at-a-glance and am asking > each > > in turn about their current memory, load, and disk space, things would > get > > slow (I'm playing with Rails caching to solve this, but that's still > seems > > like a lot of data to pull regardless) > > Yes, thats true, currently Fog will not help you much here, you would > need to query each and every hypervisor, an alternative (outside the > scope of fog and libvirt) could be to write an mcollective plugin, > which gets a list of requirements (xx cpus, yy memory, zz disks (each > with disk type and space etc) and respond with some sort of value, in > which, you could make a decision on which one is avail atm. > > > Last, if this seems like a viable endeavor, are there examples of > extending > > the API? I've got enough to go on to create some new dashboards and > models, > > but I haven't seen a clear example demonstrating how to extend the API > yet. > > its pretty easy to extend the api via a plugin, its actually identical > to adding another controller, but instead of inheriting from the > application_controller, you do it from the api controller (v1 or v2 as > you prefer). > > > > > Thank you all for such a great open source solution, the fact that I can > > whip up a plugin for a need like this so easily is incredible. If > there's > > any interest I can share the github repo as well for feedback. > > Please do share, we welcome any foreman plugin to join the foreman > github organization. > > Hope this helps, > Ohad > > [1] http://www.ovirt.org/ > [2] http://fog.io/ >

Hey,

if you want just memory, disc and load, do you really need to integrate
with individual providers? Because I plan to write tiny little
monitoring daemon that would allow to read actual basic data of guests
(memory, load, disc and maybe few other things). Once it is able to
monitor guests, it can be extended to hypervisors themselves.

I guess the only issue is the memory, because it is usually only
possible to get rough estimations of currently used memory in Linux.
This is where some integration is needed, but if you can live with that,
this could work nice. I mean for load and disk it will work just fine.

This is rather an idea I carry in my head than something real, although
I already made some notes, drawings and created few directories in my
"work" folder :slight_smile:

> In the interim, are there some examples regarding how to package
> foreman_plugins for inclusion in an rpm repository? I'm a bit of a new when
> it comes to packaging gems into rpms. I assume there's probably a specfile
> I can look at somewhere?

All our plugins are here:

https://github.com/theforeman/foreman-packaging/tree/master/rpms

Feel free to poke me with questions on packaging.

LZ

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

Hey,

last weekend I completed my prototype which is now working. Try it on
your own if it fits your needs.

I'd like to finish this and make a foreman plugin for that.

LZ

··· On Mon, Feb 03, 2014 at 10:02:44PM +0100, Lukas Zapletal wrote: > Hey, > > if you want just memory, disc and load, do you really need to integrate > with individual providers? Because I plan to write tiny little > monitoring daemon that would allow to read actual basic data of guests > (memory, load, disc and maybe few other things). Once it is able to > monitor guests, it can be extended to hypervisors themselves. > > I guess the only issue is the memory, because it is usually only > possible to get rough estimations of currently used memory in Linux. > This is where some integration is needed, but if you can live with that, > this could work nice. I mean for load and disk it will work just fine. > > This is rather an idea I carry in my head than something real, although > I already made some notes, drawings and created few directories in my > "work" folder :-) > > > In the interim, are there some examples regarding how to package > > foreman_plugins for inclusion in an rpm repository? I'm a bit of a new when > > it comes to packaging gems into rpms. I assume there's probably a specfile > > I can look at somewhere? > > All our plugins are here: > > https://github.com/theforeman/foreman-packaging/tree/master/rpms > > Feel free to poke me with questions on packaging. > > LZ > > -- > Later, > > Lukas "lzap" Zapletal > irc: lzap #theforeman > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

Okay, forgot to send the link for you:

Let me know how you like it.

LZ

··· On Mon, Feb 10, 2014 at 09:51:48PM +0100, Lukas Zapletal wrote: > Hey, > > last weekend I completed my prototype which is now working. Try it on > your own if it fits your needs. > > I'd like to finish this and make a foreman plugin for that. > > LZ > > On Mon, Feb 03, 2014 at 10:02:44PM +0100, Lukas Zapletal wrote: > > Hey, > > > > if you want just memory, disc and load, do you really need to integrate > > with individual providers? Because I plan to write tiny little > > monitoring daemon that would allow to read actual basic data of guests > > (memory, load, disc and maybe few other things). Once it is able to > > monitor guests, it can be extended to hypervisors themselves. > > > > I guess the only issue is the memory, because it is usually only > > possible to get rough estimations of currently used memory in Linux. > > This is where some integration is needed, but if you can live with that, > > this could work nice. I mean for load and disk it will work just fine. > > > > This is rather an idea I carry in my head than something real, although > > I already made some notes, drawings and created few directories in my > > "work" folder :-) > > > > > In the interim, are there some examples regarding how to package > > > foreman_plugins for inclusion in an rpm repository? I'm a bit of a new when > > > it comes to packaging gems into rpms. I assume there's probably a specfile > > > I can look at somewhere? > > > > All our plugins are here: > > > > https://github.com/theforeman/foreman-packaging/tree/master/rpms > > > > Feel free to poke me with questions on packaging. > > > > LZ > > > > -- > > Later, > > > > Lukas "lzap" Zapletal > > irc: lzap #theforeman > > > > -- > > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > > For more options, visit https://groups.google.com/groups/opt_out. > > -- > Later, > > Lukas "lzap" Zapletal > irc: lzap #theforeman > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out.


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman