Foreman-cloudstack plugin: New development, adopt to foreman 1.7/1.8, build a proper gem

Development
#1

Hello everybody.

We just started to improve the abandoned foreman-cloudstack plugin: The
auther did a good job, but there was still a lot of work left. We started
working on this plugin and already got some things going:

  • the plugin is now compatible with foreman 1.7
  • Several features have been implemented, like selecting zones and
    hypervisors, as well as service and disk offerings

What I am struggling with at the moment is to create a proper gem that
installs itself as a foreman plugin. Perhaps someone might have a look, at
the moment I do not see the mistake.

The code can be found here:
https://github.com/bytemine/foreman-cloudstack/tree/gem-install
The gem has been uploaded to rubygems.org:
https://rubygems.org/gems/foreman_cloudstack2

But no matter how I define the loading of my files inside the engine.rb I
always end up with Fog initialization failed - uninitialized constant
FogExtensions::Cloudstack and the plugin is not listed as installed plugin
within foreman.
Any idea is really appreciated!

Bye, Daniel

#2

> Hello everybody.
>
> We just started to improve the abandoned foreman-cloudstack plugin: The
> auther did a good job, but there was still a lot of work left. We
> started working on this plugin and already got some things going:

That's great! If we can help you and Brian (ddoc) maintain it better,
e.g. by moving or forking the repo over to our shared GitHub org
('theforeman') so you can maintain it together, then let us know.

> * the plugin is now compatible with foreman 1.7
> * Several features have been implemented, like selecting zones and
> hypervisors, as well as service and disk offerings
>
> What I am struggling with at the moment is to create a proper gem that
> installs itself as a foreman plugin. Perhaps someone might have a look,
> at the moment I do not see the mistake.
>
> The code can be found here:
> https://github.com/bytemine/foreman-cloudstack/tree/gem-install
> The gem has been uploaded to rubygems.org:
> https://rubygems.org/gems/foreman_cloudstack2
>
> But no matter how I define the loading of my files inside the engine.rb
> I always end up with /Fog initialization failed - uninitialized constant
> FogExtensions::Cloudstack/ and the plugin is not listed as installed
> plugin within foreman.
> Any idea is really appreciated!

I gave it a quick test locally in a source installation (which might
change the characteristics) but wasn't able to reproduce this. Do you
have a full stack trace from the error?

··· On 24/04/15 08:52, Daniel Rauer wrote:

–
Dominic Cleal
Red Hat Engineering

#3

Hi everybody.

I am wondering how the MAC and IP returned from Cloudstack after the VM has
been provisioned can be stored in foreman? Currently it just gets ignored,
but I would like to save them for the created VM in foreman. Any ideas how
to achieve this?
Still on foreman 1.7 …

Bye, Daniel

#4

Hi.

That's great! If we can help you and Brian (ddoc) maintain it better,
> e.g. by moving or forking the repo over to our shared GitHub org
> ('theforeman') so you can maintain it together, then let us know.
>

I am currently trying to keep contact with ddoc, we shared one email, but
at the moment he does not respond to my proposals or questions.
Forking the repo to the foreman-repo would be great, our github account
only reaches few people.

I gave it a quick test locally in a source installation (which might
> change the characteristics) but wasn't able to reproduce this. Do you
> have a full stack trace from the error?
>

At the moment I am setting up a fresh foreman install, will try again and
provide a stack trace.
You had the plugin up and running? What do you mean by source installation,
you took our repo and integrated the plugin manually? Could you provide
these steps to me, perhaps my engine is missing something.

Thanks for your help!
Bye, Daniel

#5

> Hi.
>
> That's great! If we can help you and Brian (ddoc) maintain it better,
> e.g. by moving or forking the repo over to our shared GitHub org
> ('theforeman') so you can maintain it together, then let us know.
>
>
> I am currently trying to keep contact with ddoc, we shared one email,
> but at the moment he does not respond to my proposals or questions.
> Forking the repo to the foreman-repo would be great, our github account
> only reaches few people.

Okay, if that's still the case after a while then we can do this. I
just added a comment to the last PR on the repo too.

> I gave it a quick test locally in a source installation (which might
> change the characteristics) but wasn't able to reproduce this. Do you
> have a full stack trace from the error?
>
>
> At the moment I am setting up a fresh foreman install, will try again
> and provide a stack trace.
> You had the plugin up and running? What do you mean by source
> installation, you took our repo and integrated the plugin manually?
> Could you provide these steps to me, perhaps my engine is missing something.

It seems I didn't! Just tried again and realised that because the
gemspec's renamed "foreman_cloudstack2" that I also needed to add
:require => 'foreman_cloudstack' to my Gemfile to ensure it tries to
load the right file.

I got the same error and fixed it by changing the autoload line in
engine.rb to:

config.autoload_paths += Dir["#{config.root}/app/models/concerns"]

Otherwise it's probably using the cwd, which usually won't be the plugin.

I also changed app/models/foreman_cloudstack/cloudstack.rb to "module
ForemanCloudstack" so it follows the usual naming convention, as that
came up as another error.

Both fixes are available here:

By a source installation, I mean checking out Foreman from source rather
than a package installation (Foreman :: Contribute) and
then adding the plugin to bundler.d/Gemfile.local.rb.

Cheers,

··· On 24/04/15 14:09, Daniel Rauer wrote:

–
Dominic Cleal
Red Hat Engineering

#6

The provided_attributes method on a compute resource is key here, it
provides a hash of Foreman host attributes to methods on the VM that
Foreman will call and set.

The methods on the VM are either provided by Fog
(lib/fog/cloudstack/models/compute/server.rb) or you can add them in
your Fog extensions:

It looks like the IP ought to work from what you have there, since
test_method's defined correctly.

You'd also need to add :mac to that hash and define a method that
retrieves the MAC address, then it should also do that and not require
the user to enter one.

··· On 09/06/15 12:17, Daniel Rauer wrote: > Hi everybody. > > I am wondering how the MAC and IP returned from Cloudstack after the VM > has been provisioned can be stored in foreman? Currently it just gets > ignored, but I would like to save them for the created VM in foreman. > Any ideas how to achieve this? > Still on foreman 1.7 ...

–
Dominic Cleal
Red Hat Engineering

#7

>
> It seems I didn't! Just tried again and realised that because the
> gemspec's renamed "foreman_cloudstack2" that I also needed to add
> :require => 'foreman_cloudstack' to my Gemfile to ensure it tries to
> load the right file.
>

I had to rename in order to be able to upload the gem to rubygems.org.

> I got the same error and fixed it by changing the autoload line in
> engine.rb to:
>
> config.autoload_paths += Dir["#{config.root}/app/models/concerns"]
>
> Otherwise it's probably using the cwd, which usually won't be the plugin.
>
> I also changed app/models/foreman_cloudstack/cloudstack.rb to "module
> ForemanCloudstack" so it follows the usual naming convention, as that
> came up as another error.
>

Thanks, just integrated your changes.

> By a source installation, I mean checking out Foreman from source rather
> than a package installation (Foreman :: Contribute) and
> then adding the plugin to bundler.d/Gemfile.local.rb.
>

I just set up a fresh install (Ubuntu 14.04, foreman 1.7.4 via
foreman-installer). Afterwards I tried installing the gem via 'gem install
foreman_cloudstack2', but it does not seem to get loaded. Then I created a
file 'bundler.d/foreman_cloudstack2.rb', inserted 'gem
"foreman_cloudstack2"' and ran 'bundle install'.
Updating files in vendor/cache

  • foreman_cloudstack2-0.1.2.gem
    Your bundle is complete!
    It was installed into ./vendor

But still no hint from the plugin to be seen in log file, nor in About
section.

Any idea what I am missing?

Thanks again, Daniel

#8

Probably the same thing as me. Bundler normally requires a file named
after the gem as an initialisation point, so in this case it'd try
loading "foreman_cloudstack2", which doesn't exist.

Try changing your bundler.d file to:

gem 'foreman_cloudstack2', :require => 'foreman_cloudstack'

Or add a lib/foreman_cloudstack2.rb to it (which can be a copy of the
original, or require the original).

··· On 24/04/15 15:14, Daniel Rauer wrote: > I just set up a fresh install (Ubuntu 14.04, foreman 1.7.4 via > foreman-installer). Afterwards I tried installing the gem via 'gem > install foreman_cloudstack2', but it does not seem to get loaded. Then I > created a file 'bundler.d/foreman_cloudstack2.rb', inserted 'gem > "foreman_cloudstack2"' and ran 'bundle install'. > > > Updatingfiles invendor/cache > *foreman_cloudstack2-0.1.2.gem > Yourbundle iscomplete! > Itwas installed into./vendor > > > > But still no hint from the plugin to be seen in log file, nor in About > section. > > Any idea what I am missing?

–
Dominic Cleal
Red Hat Engineering

#9

Hi Dominic.

··· 2015-06-09 13:29 GMT+02:00 Dominic Cleal :

The provided_attributes method on a compute resource is key here, it
provides a hash of Foreman host attributes to methods on the VM that
Foreman will call and set.

Thanks alot, that did the trick! Just created a new rubygem and pushed to
rubygems.org and github.
Daniel

#10

Hi. I ran into several problems caused by the renaming, so I renamed
everything to foreman_cloudstack2. Pushed and rebuild the gem, now it runs
fine and smooth!
I will try to ask ddoc if he is interested to merge everything and re-use
the original name.

Another problem I cannot figure out: I would like to overwrite
setUserData() in app/models/concerns/orchestration/compute.rb, but no
matter what I tried my changes from inside the plugin are not loaded, but
the original compute.rb by foreman. Am I missing something on overwriting
something from within a plugin?

Bye, Daniel

··· Am Freitag, 24. April 2015 16:17:58 UTC+2 schrieb Dominic Cleal:

Probably the same thing as me. Bundler normally requires a file named
after the gem as an initialisation point, so in this case it’d try
loading “foreman_cloudstack2”, which doesn’t exist.

#11

Thanks for working on this. Im sorry I wasn't able to be more helpful
during the development. Ill try it out soon and hopefully I can submit some
PRs!

··· On Thursday, June 11, 2015 at 1:18:52 AM UTC-7, Daniel Rauer wrote: > > Hi Dominic. > > 2015-06-09 13:29 GMT+02:00 Dominic Cleal <dcle...@redhat.com > >: > >> The provided_attributes method on a compute resource is key here, it >> provides a hash of Foreman host attributes to methods on the VM that >> Foreman will call and set. >> > > Thanks alot, that did the trick! Just created a new rubygem and pushed to > rubygems.org and github. > Daniel >
#12

You should be able to stack concerns, so rather than naming it the same
as the core one (which probably will have loaded by the time your plugin
loads), name it something else
(app/models/concerns/foreman_cloudstack/compute.rb,
ForemanCloudstack::Compute) and then in engine.rb try this:

Host::Managed.send(:include, ForemanCloudstack::Compute)

In your concern file, you can then delete everything apart from the
method(s) you want to override. I should note that this will take
effect for every compute resource type, so longer term it should be
moved into the compute resource class itself and Foreman made more
generic if needed.

··· On 27/04/15 08:21, Daniel Rauer wrote: > Am Freitag, 24. April 2015 16:17:58 UTC+2 schrieb Dominic Cleal: > > Probably the same thing as me. Bundler normally requires a file named > after the gem as an initialisation point, so in this case it'd try > loading "foreman_cloudstack2", which doesn't exist. > > > Hi. I ran into several problems caused by the renaming, so I renamed > everything to foreman_cloudstack2. Pushed and rebuild the gem, now it > runs fine and smooth! > I will try to ask ddoc if he is interested to merge everything and > re-use the original name. > > Another problem I cannot figure out: I would like to overwrite > setUserData() in app/models/concerns/orchestration/compute.rb, but no > matter what I tried my changes from inside the plugin are not loaded, > but the original compute.rb by foreman. Am I missing something on > overwriting something from within a plugin?

–
Dominic Cleal
Red Hat Engineering

#13

First, let me mention that I really appreciate all the work done on this
plugin! I'm currently upgrading my Foreman in order to take advantage of
this plugin. However, I'm having issues with 1.8.2. What is the latest
version which is compatible?

Thanks!

#14

Hi Dominic.

Thanks alot, that did the trick! Now the plugin is fully installable by
adding the gem to the Gemfile or a bundler.d-file, no more need to
overwrite foreman files.

fyi: We did a final renaming of the plugin, it is now called
"foreman_cpp_cloudstack" in reference to Citrix CloudPlatform. Do you have
plans to list Cloudstack as a supported platform based on this plugin?

Bye, Daniel

··· Am Montag, 27. April 2015 09:34:07 UTC+2 schrieb Dominic Cleal: > > On 27/04/15 08:21, Daniel Rauer wrote: > > Am Freitag, 24. April 2015 16:17:58 UTC+2 schrieb Dominic Cleal: > > > > Probably the same thing as me. Bundler normally requires a file > named > > after the gem as an initialisation point, so in this case it'd try > > loading "foreman_cloudstack2", which doesn't exist. > > > > > > Hi. I ran into several problems caused by the renaming, so I renamed > > everything to foreman_cloudstack2. Pushed and rebuild the gem, now it > > runs fine and smooth! > > I will try to ask ddoc if he is interested to merge everything and > > re-use the original name. > > > > Another problem I cannot figure out: I would like to overwrite > > setUserData() in app/models/concerns/orchestration/compute.rb, but no > > matter what I tried my changes from inside the plugin are not loaded, > > but the original compute.rb by foreman. Am I missing something on > > overwriting something from within a plugin? > > You should be able to stack concerns, so rather than naming it the same > as the core one (which probably will have loaded by the time your plugin > loads), name it something else > (app/models/concerns/foreman_cloudstack/compute.rb, > ForemanCloudstack::Compute) and then in engine.rb try this: > > Host::Managed.send(:include, ForemanCloudstack::Compute) > > In your concern file, you can then delete everything apart from the > method(s) you want to override. I should note that this will take > effect for every compute resource type, so longer term it should be > moved into the compute resource class itself and Foreman made more > generic if needed. > > -- > Dominic Cleal > Red Hat Engineering >
#15

I'd suggest adding it to the list of plugins over here:
http://projects.theforeman.org/projects/foreman/wiki/List_of_Plugins

The next thing to do will be for someone to build Debian and RPM
packages, which will make it possible for users to install it. If
you're interested, then have a look at the
https://github.com/theforeman/foreman-packaging/ repo, else perhaps me
or someone else will get to it soon!

··· On 04/05/15 12:49, Daniel Rauer wrote: > Hi Dominic. > > Thanks alot, that did the trick! Now the plugin is fully installable by > adding the gem to the Gemfile or a bundler.d-file, no more need to > overwrite foreman files. > > fyi: We did a final renaming of the plugin, it is now called > "foreman_cpp_cloudstack" in reference to Citrix CloudPlatform. Do you > have plans to list Cloudstack as a supported platform based on this plugin?

–
Dominic Cleal
Red Hat Engineering