How to deb package foreman / upgrade from 1.6.3 to custom 1.9.0

A quick synopsis:

We utilize foreman to provision our home grown customized OS to units. Our
current production foreman system is sitting at 1.6.3.

We do multiple builds a day and subsequently generate a new debian repo
(containing all of the customized OS) with each build. We have a desire to
be able to provision any given build. We COULD do this by adding an
individual OS for each build, but this would be unmanageable from an
administration perspective imo (hundreds of OSes).

So I created a foreman dev environment and was able to make custom changes
against the 1.9.0 release. I have this working fine in my dev box.

My problem now lies in upgrading the 1.6.3 foreman instance to the custom
1.9.0. As we installed the 1.6.3 with deb pkgs, my thought was to create
the debs for my 1.9.0 build and upgrade the packages. However, I am having
issues figuring out how that works. Is there a guide, doc, etc. on how
precisely to build your own foreman debs? Or is there a better solution
for what I am attempting.

Any assistance is greatly appreciated.

Thanks.

What are the changes you made to accomodate your way of working?

Foreman's best modified by writing a plugin,
Plugins - Foreman

If you still wanted to go the route of your own custom foreman spin, I
can't really help much with how to build them, but the sources are all
in https://github.com/theforeman/foreman-packaging/tree/deb/develop

··· On Fri, Aug 14, 2015 at 12:25:27PM -0700, Ryan Dyer wrote: > A quick synopsis: > > We utilize foreman to provision our home grown customized OS to units. Our > current production foreman system is sitting at 1.6.3. > > We do multiple builds a day and subsequently generate a new debian repo > (containing all of the customized OS) with each build. We have a desire to > be able to provision any given build. We COULD do this by adding an > individual OS for each build, but this would be unmanageable from an > administration perspective imo (hundreds of OSes). > > So I created a foreman dev environment and was able to make custom changes > against the 1.9.0 release. I have this working fine in my dev box. > > My problem now lies in upgrading the 1.6.3 foreman instance to the custom > 1.9.0. As we installed the 1.6.3 with deb pkgs, my thought was to create > the debs for my 1.9.0 build and upgrade the packages. However, I am having > issues figuring out how that works. Is there a guide, doc, etc. on how > precisely to build your own foreman debs? Or is there a better solution > for what I am attempting. > > Any assistance is greatly appreciated.


Best Regards,

Stephen Benjamin
Red Hat Engineering

Usually you should only upgrade one release at a time, avoid big jumps
as DB migrations may not work reliably. You may be OK to now do 1.7 to
1.9 as our unit tests say it should work, but it's untested AFAIK.

··· On 14/08/15 20:25, Ryan Dyer wrote: > A quick synopsis: > > We utilize foreman to provision our home grown customized OS to units. > Our current production foreman system is sitting at 1.6.3. > > We do multiple builds a day and subsequently generate a new debian repo > (containing all of the customized OS) with each build. We have a desire > to be able to provision any given build. We COULD do this by adding an > individual OS for each build, but this would be unmanageable from an > administration perspective imo (hundreds of OSes). > > So I created a foreman dev environment and was able to make custom > changes against the 1.9.0 release. I have this working fine in my dev > box. > > My problem now lies in upgrading the 1.6.3 foreman instance to the > custom 1.9.0. As we installed the 1.6.3 with deb pkgs, my thought was > to create the debs for my 1.9.0 build and upgrade the packages.


Dominic Cleal
Red Hat Engineering

That didn't seem to work. I found in my original tinkering with foreman
that variables with _id have a behavior to them I do not want so I created
my var without the _. I modified your example to remove the _, though
perhaps you had it there for a reason unknown to me.

engine.rb -
config.to_prepare do
begin
Host.send(:include, AddBuildId::HostExtensions) # should this be
Host or Operatingsystem ?

host_extensions.rb
module AddBuildId
module HostExtensions
extend ActiveSupport::Concern
included do
alias_method_chain :medium_uri, :buildid
end
def medium_uri_with_buildid(host, url = nil)
uri = medium_uri_without_buildid(host, url).to_s.gsub('$buildid',
host.buildid)
URI.parse(uri).normalize
end
end
end

The major change involves adding to a new entry on the edit host >
operating system called buildid (string) and modifying the
interpolate_medium_vars function to allow a $build id string to be used in
the installation media and have the buildid be properly subbed into place.

I'm all for trying this the proper plugin way, but looking at the examples
I don't see anything for modifying a view. Would I simply need to drop the
existing view and create a new one?

From a getting this implemented into our production version, it seems the
first step(s) would be to upgrade out existing installation from 1.6.3 ->
1.7 -> 1.8 -> 1.9 and then figure out how to apply the custom changes.
Does this sound correct?

Thanks!

··· On Friday, August 14, 2015 at 5:24:11 PM UTC-5, stephen wrote:

What are the changes you made to accomodate your way of working?

Foreman’s best modified by writing a plugin,
Plugins - Foreman

If you still wanted to go the route of your own custom foreman spin, I
can’t really help much with how to build them, but the sources are all
in https://github.com/theforeman/foreman-packaging/tree/deb/develop


Best Regards,

Stephen Benjamin
Red Hat Engineering

> That didn't seem to work. I found in my original tinkering with foreman
> that variables with _id have a behavior to them I do not want so I created
> my var without the _. I modified your example to remove the _, though
> perhaps you had it there for a reason unknown to me.

No, no particular reason, the alias_method_chain name doesn't need to be
anything in particular, although you're right rails might treat _id
special somewhere.

> engine.rb -
> config.to_prepare do
> begin
> Host.send(:include, AddBuildId::HostExtensions) # should this be
> Host or Operatingsystem ?

Oh definitely operating system, and I might rename it to
OperatingSystemExtensions, too.

··· On Fri, Aug 21, 2015 at 03:55:22PM -0700, Ryan Dyer wrote:

host_extensions.rb
module AddBuildId
module HostExtensions
extend ActiveSupport::Concern
included do
alias_method_chain :medium_uri, :buildid
end
def medium_uri_with_buildid(host, url = nil)
uri = medium_uri_without_buildid(host, url).to_s.gsub(’$buildid’,
host.buildid)
URI.parse(uri).normalize
end
end
end


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Best Regards,

Stephen Benjamin
Red Hat Engineering

> The major change involves adding to a new entry on the edit host >
> operating system called buildid (string) and modifying the
> interpolate_medium_vars function to allow a $build id string to be used in
> the installation media and have the buildid be properly subbed into place.
>
> I'm all for trying this the proper plugin way, but looking at the examples
> I don't see anything for modifying a view. Would I simply need to drop the
> existing view and create a new one?

You can use deface to modify an existing view. A few plugins do it,
like foreman_salt: https://github.com/theforeman/foreman_salt/tree/master/app/overrides
foreman_column_view:
https://github.com/GregSutcliffe/foreman_column_view/tree/master/app/overrides

>
> From a getting this implemented into our production version, it seems the
> first step(s) would be to upgrade out existing installation from 1.6.3 ->
> 1.7 -> 1.8 -> 1.9 and then figure out how to apply the custom changes.
> Does this sound correct?

Yea, that sounds right

··· On Mon, Aug 17, 2015 at 08:29:43AM -0700, Ryan Dyer wrote:

Thanks!

On Friday, August 14, 2015 at 5:24:11 PM UTC-5, stephen wrote:

What are the changes you made to accomodate your way of working?

Foreman’s best modified by writing a plugin,
http://projects.theforeman.org/projects/foreman/wiki/Plugins

If you still wanted to go the route of your own custom foreman spin, I
can’t really help much with how to build them, but the sources are all
in https://github.com/theforeman/foreman-packaging/tree/deb/develop


Best Regards,

Stephen Benjamin
Red Hat Engineering


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Best Regards,

Stephen Benjamin
Red Hat Engineering

Still no luck, moved host_extensions.rb to operatingsystem_extentions.rb
with contents:
module AddBuildId
module OperatingsystemExtensions
extend ActiveSupport::Concern
included do
alias_method_chain :medium_uri, :buildid
end
def medium_uri_with_buildid(host, url = nil)
uri = medium_uri_without_buildid(host, url).to_s.gsub('$buildid',
host.buildid)
URI.parse(uri).normalize
end
end
end

Modified engine.rb to contain:
config.to_prepare do
begin
Operatingsystem.send(:include,
AddBuildId::OperatingsystemExtensions)

Have an overrides/add_buildid_to_hosts.rb containing:
buildid_text = "\n <%= text_f f, :buildid, :required => false, :label =>
_(&quot;Build ID&quot;) %>\n"
Deface::Override.new(
:virtual_path => 'common/os_selection/_operatingsystem',
:name => 'add_buildid_to_hosts',
:insert_after => "erb[loud]:contains('medium_id')",
:text => buildid_text
)
Validated that this creates the updated _operatingsystem with: rake
deface:get_result[common/os_selection/_operatingsystem]

But when I click build on my host, my provisioning template still ends up
with $buildid for the value… :frowning:

What am I missing? Thank you for all the help thus far.

Ok so I believe I realized what I am missing. Though I'm not sure how to
implement. Though I have :buildid in the db, and being saved by the new
entry, there is nothing tying it into the Host class. In my pure foreman
modification I went and added the :buildid parameter, just about everywhere
(api/v[12], various models and controllers) and didn't really know where to
put it (but it worked). Can someone enlighten me to where it really should
have gone? and how to add a new parameter via the plugin?

Thanks again.

> Ok so I believe I realized what I am missing. Though I'm not sure how to
> implement. Though I have :buildid in the db, and being saved by the new
> entry, there is nothing tying it into the Host class. In my pure foreman
> modification I went and added the :buildid parameter, just about everywhere
> (api/v[12], various models and controllers) and didn't really know where to
> put it (but it worked). Can someone enlighten me to where it really should
> have gone? and how to add a new parameter via the plugin?

How did buildid get into the database? Do you write a database migration?
You would add buildid as a column on hosts, which is what ties it to the
host.

··· On Fri, Aug 21, 2015 at 04:37:33PM -0700, Ryan Dyer wrote:

Thanks again.


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Best Regards,

Stephen Benjamin
Red Hat Engineering

Thanks for the input on trying a plugin. I have successfully figured out
how to add the new field to the db and create it under the Media field
(edit host, OS). I am having issues however overwriting the methods in the
models/operatingsystems class. As per the example I have created a
models/host_extensions.rb with the following contents

module AddBuildId
module HostExtensions
extend ActiveSupport::Concern
module ClassMethods
def medium_uri(host, url = nil)
url ||= host.medium.path
medium_vars_to_uri(url, host.architecture.name,
host.operatingsystem, host.buildid)
end
def medium_vars_to_uri(url, arch, os, buildid)
URI.parse(interpolate_medium_vars(url, arch, os, buildid)).normalize
end
def interpolate_medium_vars(path, arch, os, buildid)
return "" if path.empty?
path.gsub('$arch', arch).
gsub('$major', os.major).
gsub('$minor', os.minor).
gsub('$version', os.minor.blank? ? os.major : [os.major,
os.minor].compact.join('.')).
gsub('$release', os.release_name.blank? ? '' :
os.release_name).
gsub('$buildid', buildid)
end
end
end
end

··· --------

And within the engine.rb loading it via:
config.to_prepare do
begin
Operatingsystem.send(:include, AddBuildId::HostExtensions)


but it is not doing what I want still. Any suggestions would be greatly
appreciated.

Thanks,
Ryan

yes I wrote a db migration
201508211412_add_buildid_to_host.rb
class AddBuildIdToHost < ActiveRecord::Migration
def self.up
add_column :hosts, :buildid, :string, :default => 'stable'
end
def self.down
remove_column :hosts, :buildid
end
end

··· On Friday, August 21, 2015 at 7:19:01 PM UTC-5, stephen wrote:

On Fri, Aug 21, 2015 at 04:37:33PM -0700, Ryan Dyer wrote:

Ok so I believe I realized what I am missing. Though I’m not sure how
to
implement. Though I have :buildid in the db, and being saved by the new
entry, there is nothing tying it into the Host class. In my pure
foreman
modification I went and added the :buildid parameter, just about
everywhere
(api/v[12], various models and controllers) and didn’t really know where
to
put it (but it worked). Can someone enlighten me to where it really
should
have gone? and how to add a new parameter via the plugin?

How did buildid get into the database? Do you write a database migration?
You would add buildid as a column on hosts, which is what ties it to the
host.

Thanks again.


You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/d/optout.


Best Regards,

Stephen Benjamin
Red Hat Engineering

>
> Ok so I finally got things figured out from the plugin perspective. I
>> didn't have my filenames properly matching my class name.
>>
>
I got the plugin working on my dev environment and all runs fine. But when
I build it (rake build) and then import it into my prod instance (adding it
to bundler.d, bundle install, restart apache) it shows up in my plugins but
fails to work. Debugging I see that it is not creating the new column in
the db. I've looked in all of /var/log for anything related to attempting
to add the migration. I've pushed the plugin to rubygems so if anyone
wants to see the actual changes, it's there (add_buildid). I have the
migrate script, I'm initializing the migrate stuff in the engine per the
documentation and various other plugins I see that make db changes and it
works on my dev instance. My prod server does use a pgsql instance instead
of sqlite3(dev server). Could this have anything to do with?

Any assistance would be appreciated.

Thanks,
Ryan

> Thanks for the input on trying a plugin. I have successfully figured out
> how to add the new field to the db and create it under the Media field
> (edit host, OS). I am having issues however overwriting the methods in the
> models/operatingsystems class. As per the example I have created a
> models/host_extensions.rb with the following contents
>
> module AddBuildId
> module HostExtensions
> extend ActiveSupport::Concern
> module ClassMethods
> def medium_uri(host, url = nil)
> url ||= host.medium.path
> medium_vars_to_uri(url, host.architecture.name,
> host.operatingsystem, host.buildid)
> end
> def medium_vars_to_uri(url, arch, os, buildid)
> URI.parse(interpolate_medium_vars(url, arch, os, buildid)).normalize
> end
> def interpolate_medium_vars(path, arch, os, buildid)
> return "" if path.empty?
> path.gsub('$arch', arch).
> gsub('$major', os.major).
> gsub('$minor', os.minor).
> gsub('$version', os.minor.blank? ? os.major : [os.major,
> os.minor].compact.join('.')).
> gsub('$release', os.release_name.blank? ? '' :
> os.release_name).
> gsub('$buildid', buildid)
> end
> end
> end
> end

Hi,

Glad you got the plugin going.

Part of the problem is these aren't class methods, but rather instance methods.

I would reccomend alias_method_chain[1] to extend medium_uri itself, instead of
trying to get down into the interpolate_medium_vars method.

Something like this should do:

module AddBuildId
module HostExtensions
extend ActiveSupport::Concern

included do
  alias_method_chain :medium_uri, :build_id
end

def medium_uri_with_build_id(host, url = nil)
  medium_uri_without_build_id(host, url).gsub(&#39;$buildid&#39;, host.buildid)
end

end
end

[1] http://apidock.com/rails/v3.2.8/Module/alias_method_chain

··· On Fri, Aug 21, 2015 at 02:51:35PM -0700, Ryan Dyer wrote:

And within the engine.rb loading it via:
config.to_prepare do
begin
Operatingsystem.send(:include, AddBuildId::HostExtensions)


but it is not doing what I want still. Any suggestions would be greatly
appreciated.

Thanks,
Ryan


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Best Regards,

Stephen Benjamin
Red Hat Engineering

Have you run rake db:migrate? There's no mechanism apart from in
deb/RPM packages of plugins to migrate automatically.

Running rake db:migrate:status will also show you which migrations are
known and have been run, so you'll see if it's up/down.

··· On 02/09/15 03:06, Ryan Dyer wrote: > Ok so I finally got things figured out from the plugin > perspective. I didn't have my filenames properly matching my > class name. > > > I got the plugin working on my dev environment and all runs fine. But > when I build it (rake build) and then import it into my prod instance > (adding it to bundler.d, bundle install, restart apache) it shows up in > my plugins but fails to work. Debugging I see that it is not creating > the new column in the db.


Dominic Cleal
dominic@cleal.org

> > Thanks for the input on trying a plugin. I have successfully figured out
> > how to add the new field to the db and create it under the Media field
> > (edit host, OS). I am having issues however overwriting the methods in the
> > models/operatingsystems class. As per the example I have created a
> > models/host_extensions.rb with the following contents
> >
> > module AddBuildId
> > module HostExtensions
> > extend ActiveSupport::Concern
> > module ClassMethods
> > def medium_uri(host, url = nil)
> > url ||= host.medium.path
> > medium_vars_to_uri(url, host.architecture.name,
> > host.operatingsystem, host.buildid)
> > end
> > def medium_vars_to_uri(url, arch, os, buildid)
> > URI.parse(interpolate_medium_vars(url, arch, os, buildid)).normalize
> > end
> > def interpolate_medium_vars(path, arch, os, buildid)
> > return "" if path.empty?
> > path.gsub('$arch', arch).
> > gsub('$major', os.major).
> > gsub('$minor', os.minor).
> > gsub('$version', os.minor.blank? ? os.major : [os.major,
> > os.minor].compact.join('.')).
> > gsub('$release', os.release_name.blank? ? '' :
> > os.release_name).
> > gsub('$buildid', buildid)
> > end
> > end
> > end
> > end
>
> Hi,
>
> Glad you got the plugin going.
>
> Part of the problem is these aren't class methods, but rather instance methods.
>
> I would reccomend alias_method_chain[1] to extend medium_uri itself, instead of
> trying to get down into the interpolate_medium_vars method.
>
> Something like this should do:
>
> module AddBuildId
> module HostExtensions
> extend ActiveSupport::Concern
>
> included do
> alias_method_chain :medium_uri, :build_id
> end
>
> def medium_uri_with_build_id(host, url = nil)
> medium_uri_without_build_id(host, url).gsub('$buildid', host.buildid)
> end
> end
> end

Oh, sorry, small change - my gsub won't work because it's a URI.

module AddBuildId
module HostExtensions
extend ActiveSupport::Concern

included do
  alias_method_chain :medium_uri, :build_id
end

def medium_uri_with_build_id(host, url = nil)
  uri = medium_uri_without_build_id(host, url).to_s.gsub(&#39;$buildid&#39;, host.buildid)
  URI.parse(uri).normalize
end

end
end

··· On Fri, Aug 21, 2015 at 06:24:29PM -0400, Stephen Benjamin wrote: > On Fri, Aug 21, 2015 at 02:51:35PM -0700, Ryan Dyer wrote:

[1] http://apidock.com/rails/v3.2.8/Module/alias_method_chain


And within the engine.rb loading it via:
config.to_prepare do
begin
Operatingsystem.send(:include, AddBuildId::HostExtensions)


but it is not doing what I want still. Any suggestions would be greatly
appreciated.

Thanks,
Ryan


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Best Regards,

Stephen Benjamin
Red Hat Engineering


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Best Regards,

Stephen Benjamin
Red Hat Engineering

That was the piece I was missing. Thanks!

Is there something we can do to clarify this step in the plugin
instructions here:
How to Create a Plugin - Foreman ?
'Then you can use rake db:migrate in your app directly.', imo, implies that
the application will somehow utilize the migrate, not that I need to run
this on the host in order for the plugin to work.

Thanks for all your assistance!
Ryan

··· On Wednesday, September 2, 2015 at 1:57:29 AM UTC-5, Dominic Cleal wrote:

On 02/09/15 03:06, Ryan Dyer wrote:

    Ok so I finally got things figured out from the plugin 
    perspective.  I didn't have my filenames properly matching my 
    class name. 

I got the plugin working on my dev environment and all runs fine. But
when I build it (rake build) and then import it into my prod instance
(adding it to bundler.d, bundle install, restart apache) it shows up in
my plugins but fails to work. Debugging I see that it is not creating
the new column in the db.

Have you run rake db:migrate? There’s no mechanism apart from in
deb/RPM packages of plugins to migrate automatically.

Running rake db:migrate:status will also show you which migrations are
known and have been run, so you’ll see if it’s up/down.


Dominic Cleal
dom...@cleal.org <javascript:>