Notes from meeting with Puppet community at CfgmgmtCamp - fixing Puppet 4

Development
1

Hi all,

(TL;DR? Action items are at the bottom :P)

Among the many many things that happened last week (trip reports to follow
:P), a few of us sat down with some of the Puppet community to discuss the
current Puppet 4 woes in Foreman, and how we can potentially work together
better going forward.

Present for our side were:

  • Myself (gwmngilfen)
  • Dmitri (witlessb)
  • Marek (mhulan / ares)
  • Ewoud (ewoud)

Apologies for not being able to name everyone on the Puppet side - I know
Eric Sorenson (eric0), vstone and zipkid were there, and some others.

We started out with general discussion and an outline of the issues - why
the Proxy needs to load the puppet libs, what Kafo needs them for, and so
on.

Getting into specifics, I'll go by topic:

  • Kafo / installer support

As previously discussed on this list, we can't use any kind of API service
for the installer, as by definition nothing is running at install time. We
asked if there was any plan or chance of releasing the parser as a
standalone gem that we could use.

Turns out there is - puppet-strings is the lib, and it can emit JSON
regarding the classes parsed. Marek felt that this was a viable approach,
but the amount of work is still to be determined. Certainly there will be
some packaging at the least.

It seemed Puppet are open to collaboration on puppet-strings if we need
more from it - certainly we discussed if it would be possible to make some
of the dependencies for puppet-strings optional, as we only need the JSON
from it. No firm decisions there yet, but there's active work on the Puppet
side on this gem. There is a need to release the gem to Rubygems so we can
package it for our installer though.

This also opens up the potential to cache the JSON for the installer
modules at packaging time, for a nice decrease in execution time of the
installer itself.

I'll let Marek reply with further thoughts on how hard this will be, but it
sounded positive to me, albeit the biggest piece of the work.

  • Smart proxy

Dmitri felt that for the proxy, puppet-strings was probably inferior to
using the new Resources API in the Puppetserver. We already have code in
place for this, so it seemed sensible to go forwards from there, especially
in regard to how/when we try to load the puppet libs - there's no need to
if we're wholly relying on the API.

Dmitri felt that this was relatively easy from our side, and I believe he's
already hacking on it. I'll leave it to him to comment further. He did
request that the API be looked at for ways to improve it's efficiency, and
to add an etag for caching to both the environments and resources APIs
(jira #server-1111)

  • Puppet modules

Ewoud felt the puppet modules themselves are largely Puppet 4 compliant
already - and as a Puppet 4 user myself, I concur. We do have a few small
issues on us though:

  • Moving the report processor to lib (I've opened a PR to discuss this)
  • Improving our support for puppetserver, especially on java options
    (underway in a PR)
  • Update community-templates to use the PC1 repos
  • Fixing the smart_proxy provider which registers the proxy

Regarding the last item, the issue is that we rely on apipie_bindings in
our provider, which can't be loaded in the Puppet 4 ruby stack.

There was a discussion about the merits of apipie vs plain API use in the
providers. Given the difficulties packaging apipie for the puppet stack,
and how to load it, vs the need for occasional fixes to the providers if
the API changes, it was felt that it's probably the least work to drop
apipie from our puppet modules.

We had an offer from @liamjbennett to contribute a near-complete set of
pure api-only types and providers for most objects in Foreman, which would
be a big benefit to the community. Much appreciated!

  • Ongoing collaboration

We were reminded that Puppet hold a weekly dev hangout, which we'd be
welcome to join if we experience issues in the future - details in the
#puppet-dev IRC topic. Eric and I also agreed to be points-of-contact for
each other if there are blockers or things that come up, so do let me know
if I can help push any issues forwards.

Once the Resources API and and puppet-strings are in progress, it's
probably fairly easy to test against HEAD of those things to spot
upcoming issues. We can look into this later.

There was also a brief discussion of maybe covering/promoting a
Puppet4-compatible release of Foreman in a joint manner - blog posts,
social media, etc. I'm open to that.

  • Conclusion

Phew, quite the wall of text! Overall I got a good feeling from the meeting

  • we all want to see this fixed. Here's a summary of the action items I
    heard - speak up if I missed something or got something wrong:
  • Puppet to release puppet-strings as a gem
    • Further work there to be determined, possibly around dependencies
  • Puppet to add etag to the environments and resources APIs
  • Liam to send in PR(s) for the types/providers
    • Need someone to integrate them afterwards
  • Marek to detail Kafo update workload
  • Dmitri to update the proxy
  • Greg to look at the report processor and community-templates
  • Greg & Eric to figure out joint promotion once the work is complete
  • Greg & Eric to keep in touch going forwards

Cheers!

··· -- Greg IRC: gwmngilfen
2

> * Kafo / installer support
>
> As previously discussed on this list, we can't use any kind of API
> service for the installer, as by definition nothing is running at
> install time. We asked if there was any plan or chance of releasing the
> parser as a standalone gem that we could use.
>
> Turns out there is - puppet-strings is the lib, and it can emit JSON
> regarding the classes parsed. Marek felt that this was a viable
> approach, but the amount of work is still to be determined. Certainly
> there will be some packaging at the least.

Feature #7848: YARD / Puppet Strings support - Kafo - Foreman is the existing ticket for this.

> It seemed Puppet are open to collaboration on puppet-strings if we need
> more from it - certainly we discussed if it would be possible to make
> some of the dependencies for puppet-strings optional, as we only need
> the JSON from it. No firm decisions there yet, but there's active work
> on the Puppet side on this gem. There is a need to release the gem to
> Rubygems so we can package it for our installer though.

Seems preferable to me for it to be added to puppet-agent or similar
(file a CPR ticket?), or we package the existing module (as a module
not a gem) and execute the face.

Using a gem might be harder to package, as it would need to be installed
into the AIO environment anyway.

··· On 11/02/16 09:22, Greg Sutcliffe wrote:

–
Dominic Cleal
dominic@cleal.org

3

> We had an offer from @liamjbennett to contribute a near-complete set of
> pure api-only types and providers for most objects in Foreman, which would
> be a big benefit to the community. Much appreciated!
>

Here's a link to his slides from Cfgmgmt showing some of the work they've
done: https://speakerdeck.com/liamjbennett/puppet-for-services?slide=32

Slide 33 shows it in use.

4

Hi all - one month on from cfgmgmtcamp (time flies!) and I wanted to check in
on a few items for follow-up; comments inline below.

> It seemed Puppet are open to collaboration on puppet-strings if we need
> more from it - certainly we discussed if it would be possible to make some
> of the dependencies for puppet-strings optional, as we only need the JSON
> from it. No firm decisions there yet, but there's active work on the Puppet
> side on this gem. There is a need to release the gem to Rubygems so we can
> package it for our installer though.

From the thread about this on foreman-dev it seemed like maybe this isn't a
requirement? It's not done yet sadly, but it will likely happen before the
other alternative, which is bundling it inside the all-in-one puppet package.

> Dmitri felt that this was relatively easy from our side, and I believe he's
> already hacking on it. I'll leave it to him to comment further. He did
> request that the API be looked at for ways to improve it's efficiency, and
> to add an etag for caching to both the environments and resources APIs
> (jira #server-1111)

FYI the puppet-server release that contains this (2.3.0) is headed out the
door 17 March 2016.

> We had an offer from @liamjbennett to contribute a near-complete set of
> pure api-only types and providers for most objects in Foreman, which would
> be a big benefit to the community. Much appreciated!

Has there been any progress on this front? Anything I can help with?

Cheers
eric0

Eric Sorenson - eric.sorenson@puppetlabs.com - freenode #puppet: eric0
puppet platform // coffee // techno // bicycles

··· On Thu, 11 Feb 2016, Greg Sutcliffe wrote:
5

That's true - I'd forgotten it was a face. It does seem like including the
module would work, bu in any case, Marek was researching how to make use of
it, so I'll leave it to him to comment.

··· On 11 February 2016 at 09:40, Dominic Cleal wrote: > > Seems preferable to me for it to be added to puppet-agent or similar > (file a CPR ticket?), or we package the existing module (as a *module* > not a gem) and execute the face. > > Using a gem might be harder to package, as it would need to be installed > into the AIO environment anyway.

–
Greg
IRC: gwmngilfen

6

There’s a PR in smart-proxy that removes puppet dependency when
running in puppet 4.0 (and higher) environment:
https://github.com/theforeman/smart-proxy/pull/376.

ATM it uses an older api (resource_types) to retrieve puppet classes
however, as api https://tickets.puppetlabs.com/browse/SERVER-1111
refers to hasn’t been released into production yet (I don’t think?).

-d

··· On Thu, Feb 11, 2016 at 11:11 AM, Greg Sutcliffe wrote: > >> We had an offer from @liamjbennett to contribute a near-complete set of >> pure api-only types and providers for most objects in Foreman, which would >> be a big benefit to the community. Much appreciated! > > > Here's a link to his slides from Cfgmgmt showing some of the work they've > done: https://speakerdeck.com/liamjbennett/puppet-for-services?slide=32 > > Slide 33 shows it in use. > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.
7

Yesterday I had a look at our puppet modules. On the bright side, we did
solve our reporting support and puppetserver 2 is also properly
supported. That means we only need need to fix the smartproxy
registration.

@liamjbennett is based on a resource class[1] and is mostly compatible
with our type, so extending his provider to match should be mostly
straight forward.

The is a bigger issue is that his types/providers don't solve our main
problem: we need gems and don't want to install them at runtime since
users want to every dependency packaged in RPMs/debs. Currently we have
all gems we need in the installer packaged for the system ruby. That
won't work if puppet uses a bundled ruby.

Implementing plain REST API calls without our apipie bindings removes
the need for one gem, it still leaves us with an oauth dependency inside
the AIO install. Right now I don't know how to move forward.

[1] https://github.com/opentable/puppet-foreman/blob/97a03006c8b75d223aa635d67accbf7ef72b6a4f/lib/puppet_x/theforeman/resource.rb

··· On Tue, Mar 01, 2016 at 10:35:58AM -0800, Eric Sorenson wrote: > Hi all - one month on from cfgmgmtcamp (time flies!) and I wanted to check > in on a few items for follow-up; comments inline below. > > On Thu, 11 Feb 2016, Greg Sutcliffe wrote: > > >We had an offer from @liamjbennett to contribute a near-complete set of > >pure api-only types and providers for most objects in Foreman, which would > >be a big benefit to the community. Much appreciated! > > Has there been any progress on this front? Anything I can help with?
8

> Using a gem might be harder to package, as it would need to be installed
> into the AIO environment anyway.

We've checked dependencies, we'd need to package only puppet-strings, puppet,
hiera gems into our ruby stack. Which means we'd have to install puppet gem
twice - once in our stack, second in AIO. Are there any concerns with this?
If we have our own packages for these gems we don't have to install AIO to just
parse manifests.

> That's true - I'd forgotten it was a face. It does seem like including the
> module would work, bu in any case, Marek was researching how to make use of
> it, so I'll leave it to him to comment.

I quickly looked if we could use puppet 4 code directly but I didn't have
a chance to check puppet-strings entirely. I'll try to find some time next
week to do some research. I expect some features of kafo might be hard to
achieve with strings e.g. validations based on called functions. Other things
not related but that could also get tricky - default values, progress bar.

··· -- Marek

----- Original Message -----
From: “Greg Sutcliffe” greg.sutcliffe@gmail.com
To: “Foreman” foreman-dev@googlegroups.com
Sent: Thursday, February 11, 2016 10:49:03 AM
Subject: Re: [foreman-dev] Notes from meeting with Puppet community at CfgmgmtCamp - fixing Puppet 4

On 11 February 2016 at 09:40, Dominic Cleal dominic@cleal.org wrote:

Seems preferable to me for it to be added to puppet-agent or similar
(file a CPR ticket?), or we package the existing module (as a module
not a gem) and execute the face.

Using a gem might be harder to package, as it would need to be installed
into the AIO environment anyway.

That’s true - I’d forgotten it was a face. It does seem like including the
module would work, bu in any case, Marek was researching how to make use of
it, so I’ll leave it to him to comment.

Greg
IRC: gwmngilfen

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

9

Would it be possible to do the same for puppet <= 4.0? What's the lowest version
of puppet we could do it? This would help us greatly on moving the smart proxy
away from ruby 1.8

– Ivan

··· ----- Original Message ----- > There’s a PR in smart-proxy that removes puppet dependency when > running in puppet 4.0 (and higher) environment: > https://github.com/theforeman/smart-proxy/pull/376. > > ATM it uses an older api (resource_types) to retrieve puppet classes > however, as api https://tickets.puppetlabs.com/browse/SERVER-1111 > refers to hasn’t been released into production yet (I don’t think?). > > -d > > On Thu, Feb 11, 2016 at 11:11 AM, Greg Sutcliffe > wrote: > > > >> We had an offer from @liamjbennett to contribute a near-complete set of > >> pure api-only types and providers for most objects in Foreman, which would > >> be a big benefit to the community. Much appreciated! > > > > > > Here's a link to his slides from Cfgmgmt showing some of the work they've > > done: https://speakerdeck.com/liamjbennett/puppet-for-services?slide=32 > > > > Slide 33 shows it in use. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to foreman-dev+unsubscribe@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >
10

Hmm, that is tricky. I am asking around other projects that are in a
similar situation with their packaging (like chef omnibus and
logstash/elastic) how they handle it. Is there a workaround that's not too
horrible but can keep us moving forward until we come up with a better
solution? (Or is oauth literally the only gem in this category? if so that
is probably possible to absorb

–eric0

··· On Tuesday, March 1, 2016 at 1:27:16 PM UTC-8, Ewoud Kohl van Wijngaarden wrote: > > The is a bigger issue is that his types/providers don't solve our main > problem: we need gems and don't want to install them at runtime since > users want to every dependency packaged in RPMs/debs. Currently we have > all gems we need in the installer packaged for the system ruby. That > won't work if puppet uses a bundled ruby. > > Implementing plain REST API calls without our apipie bindings removes > the need for one gem, it still leaves us with an oauth dependency inside > the AIO install. Right now I don't know how to move forward. > > [1] > https://github.com/opentable/puppet-foreman/blob/97a03006c8b75d223aa635d67accbf7ef72b6a4f/lib/puppet_x/theforeman/resource.rb >
11

>> Hi all - one month on from cfgmgmtcamp (time flies!) and I wanted to check
>> in on a few items for follow-up; comments inline below.
>>
>>
>>> We had an offer from @liamjbennett to contribute a near-complete set of
>>> pure api-only types and providers for most objects in Foreman, which would
>>> be a big benefit to the community. Much appreciated!
>>
>> Has there been any progress on this front? Anything I can help with?
>
> Yesterday I had a look at our puppet modules. On the bright side, we did
> solve our reporting support and puppetserver 2 is also properly
> supported. That means we only need need to fix the smartproxy
> registration.
>
> @liamjbennett is based on a resource class[1] and is mostly compatible
> with our type, so extending his provider to match should be mostly
> straight forward.
>
> [1]

I've opened a pull request at
https://github.com/theforeman/puppet-foreman/pull/432 which adds a new
provider based largely on the code above. I've refactored it
considerably to fix a few problems and make it more testable, but the
concept's much the same.

> The is a bigger issue is that his types/providers don't solve our main
> problem: we need gems and don't want to install them at runtime since
> users want to every dependency packaged in RPMs/debs. Currently we have
> all gems we need in the installer packaged for the system ruby. That
> won't work if puppet uses a bundled ruby.
>
> Implementing plain REST API calls without our apipie bindings removes
> the need for one gem, it still leaves us with an oauth dependency inside
> the AIO install. Right now I don't know how to move forward.

I've created a puppet-agent-oauth package to ship the oauth gem
dependency, which installs it straight into the AIO agent environment.
The RPM version's at
https://github.com/theforeman/foreman-packaging/pull/1095 and I'm
working on the Debian equivalent.

This reduces the number of gems needed in the environment significantly,
and means we only need to maintain a single gem/package.

··· On 01/03/16 21:27, Ewoud Kohl van Wijngaarden wrote: > On Tue, Mar 01, 2016 at 10:35:58AM -0800, Eric Sorenson wrote: >> On Thu, 11 Feb 2016, Greg Sutcliffe wrote:

–
Dominic Cleal
dominic@cleal.org

12

Packaging Puppet as a dependency seems like a large commitment, one I'd
really like to avoid adding. It's also likely to add confusion for
users, as it did in the past when we shipped an RPM of Puppet as an
internal dependency of Foreman (ruby193-puppet).

If the user's installing Puppet using regular packages to run the
installer, either from their distro or the vendor's AIO packages, isn't
that enough?

··· On 12/02/16 08:17, Marek Hulan wrote: >> > Using a gem might be harder to package, as it would need to be installed >> > into the AIO environment anyway. > > We've checked dependencies, we'd need to package only puppet-strings, puppet, > hiera gems into our ruby stack. Which means we'd have to install puppet gem > twice - once in our stack, second in AIO. Are there any concerns with this? > If we have our own packages for these gems we don't have to install AIO to just > parse manifests.

–
Dominic Cleal
dominic@cleal.org

13

Puppet 4.0 was the first release that introduced api to support find
and search operations on classes, earlier versions won’t work (or
rather have to rely on built-in class parsers).

-d

··· On Fri, Feb 12, 2016 at 8:34 AM, Ivan Necas wrote: > Would it be possible to do the same for puppet <= 4.0? What's the lowest version > of puppet we could do it? This would help us greatly on moving the smart proxy > away from ruby 1.8 > > -- Ivan > > ----- Original Message ----- >> There’s a PR in smart-proxy that removes puppet dependency when >> running in puppet 4.0 (and higher) environment: >> https://github.com/theforeman/smart-proxy/pull/376. >> >> ATM it uses an older api (resource_types) to retrieve puppet classes >> however, as api https://tickets.puppetlabs.com/browse/SERVER-1111 >> refers to hasn’t been released into production yet (I don’t think?). >> >> -d
14

Hi,

> > The is a bigger issue is that his types/providers don't solve our main
> > problem: we need gems and don't want to install them at runtime since
> > users want to every dependency packaged in RPMs/debs. Currently we have
> > all gems we need in the installer packaged for the system ruby. That
> > won't work if puppet uses a bundled ruby.
> >
> > Implementing plain REST API calls without our apipie bindings removes
> > the need for one gem, it still leaves us with an oauth dependency inside
> > the AIO install. Right now I don't know how to move forward.
>
> Hmm, that is tricky. I am asking around other projects that are in a
> similar situation with their packaging (like chef omnibus and
> logstash/elastic) how they handle it. Is there a workaround that's not too
> horrible but can keep us moving forward until we come up with a better
> solution? (Or is oauth literally the only gem in this category? if so that
> is probably possible to absorb

Would it be possible to have (for AIO Puppet only, that would be done by
puppet-foreman) a RPM/DEB package that's containing the .gem file for
oauth (or, if we just leave everything as-is, the .gem files of
apipie-bindings and its dependencies) in, let's say, /opt/foreman/xyz
and use the puppet_gem provider to install these gems from the provided
files (thus also without internet access) into the AIO Ruby environment?

Regards

··· On Tue, Mar 15, 2016 at 10:17:29AM -0700, Eric Sorenson wrote: -- Michael Moll
15

> >> Hi all - one month on from cfgmgmtcamp (time flies!) and I wanted to check
> >> in on a few items for follow-up; comments inline below.
> >>
> >>
> >>> We had an offer from @liamjbennett to contribute a near-complete set of
> >>> pure api-only types and providers for most objects in Foreman, which would
> >>> be a big benefit to the community. Much appreciated!
> >>
> >> Has there been any progress on this front? Anything I can help with?
> >
> > Yesterday I had a look at our puppet modules. On the bright side, we did
> > solve our reporting support and puppetserver 2 is also properly
> > supported. That means we only need need to fix the smartproxy
> > registration.
> >
> > @liamjbennett is based on a resource class[1] and is mostly compatible
> > with our type, so extending his provider to match should be mostly
> > straight forward.
> >
> > [1]
> https://github.com/opentable/puppet-foreman/blob/97a03006c8b75d223aa635d67accbf7ef72b6a4f/lib/puppet_x/theforeman/resource.rb
>
> I've opened a pull request at
> https://github.com/theforeman/puppet-foreman/pull/432 which adds a new
> provider based largely on the code above. I've refactored it
> considerably to fix a few problems and make it more testable, but the
> concept's much the same.

Thanks to Dominics and Michaels efforts this is now merged.

> > The is a bigger issue is that his types/providers don't solve our main
> > problem: we need gems and don't want to install them at runtime since
> > users want to every dependency packaged in RPMs/debs. Currently we have
> > all gems we need in the installer packaged for the system ruby. That
> > won't work if puppet uses a bundled ruby.
> >
> > Implementing plain REST API calls without our apipie bindings removes
> > the need for one gem, it still leaves us with an oauth dependency inside
> > the AIO install. Right now I don't know how to move forward.
>
> I've created a puppet-agent-oauth package to ship the oauth gem
> dependency, which installs it straight into the AIO agent environment.
> The RPM version's at
> https://github.com/theforeman/foreman-packaging/pull/1095 and I'm
> working on the Debian equivalent.
>
> This reduces the number of gems needed in the environment significantly,
> and means we only need to maintain a single gem/package.

https://github.com/theforeman/foreman-packaging/pull/1096 was the PR for
the debs. Again thanks to Dominics and Michaels efforts both RPMs and
Debs were merged.

The current status is that we still have some open PRs to improve the
support. https://github.com/theforeman/puppet-foreman_proxy/pull/232
is the last blocker that I know of and it's waiting for tests to pass.
Hopefully this will result in some nightlies we can test.

··· On Fri, Apr 15, 2016 at 01:08:35PM +0100, Dominic Cleal wrote: > On 01/03/16 21:27, Ewoud Kohl van Wijngaarden wrote: > > On Tue, Mar 01, 2016 at 10:35:58AM -0800, Eric Sorenson wrote: > >> On Thu, 11 Feb 2016, Greg Sutcliffe wrote:
16

Could the smart proxy use kafo_parsers as an optional parser for this?

··· On Fri, Feb 12, 2016 at 08:55:25AM +0000, Dmitri Dolguikh wrote: > Puppet 4.0 was the first release that introduced api to support find > and search operations on classes, earlier versions won’t work (or > rather have to rely on built-in class parsers). > > -d > > On Fri, Feb 12, 2016 at 8:34 AM, Ivan Necas wrote: > > Would it be possible to do the same for puppet <= 4.0? What's the lowest version > > of puppet we could do it? This would help us greatly on moving the smart proxy > > away from ruby 1.8 > > > > -- Ivan > > > > ----- Original Message ----- > >> There’s a PR in smart-proxy that removes puppet dependency when > >> running in puppet 4.0 (and higher) environment: > >> https://github.com/theforeman/smart-proxy/pull/376. > >> > >> ATM it uses an older api (resource_types) to retrieve puppet classes > >> however, as api https://tickets.puppetlabs.com/browse/SERVER-1111 > >> refers to hasn’t been released into production yet (I don’t think?). > >> > >> -d > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.
17

> >> > Using a gem might be harder to package, as it would need to be
> >> > installed
> >> > into the AIO environment anyway.
> >
> > We've checked dependencies, we'd need to package only puppet-strings,
> > puppet, hiera gems into our ruby stack. Which means we'd have to install
> > puppet gem twice - once in our stack, second in AIO. Are there any
> > concerns with this? If we have our own packages for these gems we don't
> > have to install AIO to just parse manifests.
>
> Packaging Puppet as a dependency seems like a large commitment, one I'd
> really like to avoid adding. It's also likely to add confusion for
> users, as it did in the past when we shipped an RPM of Puppet as an
> internal dependency of Foreman (ruby193-puppet).

Makes sense, it would be great if we find better option.

> If the user's installing Puppet using regular packages to run the
> installer, either from their distro or the vendor's AIO packages, isn't
> that enough?

I'm not sure how would we know how to package the module. Or should we have
two versions, one for AIO and second for distro? Or is there one place where
puppet loads modules from not matter how it's installed? Also I'm not sure how
we would detect which puppet binary that should be used. I guess we will have
to check common installation paths?

I pushed a PoC of puppet strings based parser at [1]. We could probably add
third parser with another approach that would read the information from JSON
file directly (if it exist). We could generate the JSON during foreman-
installer package build process so we wouldn't need to install the string
module at all. So during build we'd use the parser from [1] and during
foreman-installer we'd use the JSON parser.

[1] https://github.com/theforeman/kafo_parsers/pull/13

··· On Friday 12 of February 2016 08:24:47 Dominic Cleal wrote: > On 12/02/16 08:17, Marek Hulan wrote:

–
Marek

18

This should work, the puppet_gem provider should be able to install from
a local gem file. It'd also be possible to run the gem install command
from the package postinstall.

In either case the package could even be installed during the Puppet run
in the same way we currently install apipie-bindings.

··· On 17/03/16 00:12, Michael Moll wrote: > Hi, > > On Tue, Mar 15, 2016 at 10:17:29AM -0700, Eric Sorenson wrote: >>> The is a bigger issue is that his types/providers don't solve our main >>> problem: we need gems and don't want to install them at runtime since >>> users want to every dependency packaged in RPMs/debs. Currently we have >>> all gems we need in the installer packaged for the system ruby. That >>> won't work if puppet uses a bundled ruby. >>> >>> Implementing plain REST API calls without our apipie bindings removes >>> the need for one gem, it still leaves us with an oauth dependency inside >>> the AIO install. Right now I don't know how to move forward. >> >> Hmm, that is tricky. I am asking around other projects that are in a >> similar situation with their packaging (like chef omnibus and >> logstash/elastic) how they handle it. Is there a workaround that's not too >> horrible but can keep us moving forward until we come up with a better >> solution? (Or is oauth literally the only gem in this category? if so that >> is probably possible to absorb > > Would it be possible to have (for AIO Puppet only, that would be done by > puppet-foreman) a RPM/DEB package that's containing the .gem file for > oauth (or, if we just leave everything as-is, the .gem files of > apipie-bindings and its dependencies) in, let's say, /opt/foreman/xyz > and use the puppet_gem provider to install these gems from the provided > files (thus also without internet access) into the AIO Ruby environment?

–
Dominic Cleal
dominic@cleal.org

19

I think that's the last update to the modules for now, but there are a
few PRs for Kafo that I'm still working on so it'll run with AIO. You
should be able to use puppet apply for the time being.

··· On 18/04/16 13:25, Ewoud Kohl van Wijngaarden wrote: > The current status is that we still have some open PRs to improve the > support. https://github.com/theforeman/puppet-foreman_proxy/pull/232 > is the last blocker that I know of and it's waiting for tests to pass. > Hopefully this will result in some nightlies we can test.

–
Dominic Cleal
dominic@cleal.org

20

Since you invoke the strings binary as opposed using to the gem, you
avoid the packaging issue. You just need to ensure puppetlabs-strings is
installed in AIO but maybe we can convince puppetlabs to ship it by
default.

Using a pre-generated JSON-file could still be a good idea and improve
runtime performance.

··· On Wed, Feb 24, 2016 at 09:43:07AM +0100, Marek Hulán wrote: > On Friday 12 of February 2016 08:24:47 Dominic Cleal wrote: > > On 12/02/16 08:17, Marek Hulan wrote: > > >> > Using a gem might be harder to package, as it would need to be > > >> > installed > > >> > into the AIO environment anyway. > > > > > > We've checked dependencies, we'd need to package only puppet-strings, > > > puppet, hiera gems into our ruby stack. Which means we'd have to install > > > puppet gem twice - once in our stack, second in AIO. Are there any > > > concerns with this? If we have our own packages for these gems we don't > > > have to install AIO to just parse manifests. > > > > Packaging Puppet as a dependency seems like a large commitment, one I'd > > really like to avoid adding. It's also likely to add confusion for > > users, as it did in the past when we shipped an RPM of Puppet as an > > internal dependency of Foreman (ruby193-puppet). > > Makes sense, it would be great if we find better option. > > > If the user's installing Puppet using regular packages to run the > > installer, either from their distro or the vendor's AIO packages, isn't > > that enough? > > I'm not sure how would we know how to package the module. Or should we have > two versions, one for AIO and second for distro? Or is there one place where > puppet loads modules from not matter how it's installed? Also I'm not sure how > we would detect which puppet binary that should be used. I guess we will have > to check common installation paths? > > I pushed a PoC of puppet strings based parser at [1]. We could probably add > third parser with another approach that would read the information from JSON > file directly (if it exist). We could generate the JSON during foreman- > installer package build process so we wouldn't need to install the string > module at all. So during build we'd use the parser from [1] and during > foreman-installer we'd use the JSON parser. > > [1] https://github.com/theforeman/kafo_parsers/pull/13