Starting with puppet 4.0, one of the the possible installation options
is to use Puppetlab’s all-in-one (AIO) packages. These come with their
own run-time environment – ruby vm (currently 2.1.0), gems, native
extensions, puppet-agent and puppet-server packages. More information
about the fs layout of AIO packages can be found at [1].
Using AIO puppet package with smart-proxy.
Using AIO-packaged puppet is pretty starightforward: $LOAD_PATH needs
to be prepended with directories in /opt/puppetlabs/puppet/lib/ruby (I
don’t know if all of them are needed; some contain dynamic libraries
used by native extensions used in puppet’s depdendencies). Gem.path
needs to include AIO environment’s gem directory
(/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0.
The main drawback is that system ruby version needs to be sufficiently
close to AIO environment one, or some gems will fail to load (I’m
getting “incompatible library version” error).
Another approach is to rely on puppet API. Currently we require puppet
gem for our puppet class parser. Starting with puppet api 3.0
‘resource_types’ endpoint is available, which allows for retrieval of
environment’s puppet classes and their parameters. See [2] for some
examples.
It appears there are some issues with access controls for that API
point, although they are easily fixed by updating puppet’s auth.conf.
I think using API is a much more attractive way of interfacing with
puppet: we won’t be constrained by ruby vm version, will drop puppet
dependency, and will eventually get rid of puppet parser code
(although we’ll need to keep it for a while).
> Using AIO-packaged puppet is pretty starightforward: $LOAD_PATH needs
> to be prepended with directories in /opt/puppetlabs/puppet/lib/ruby (I
> don’t know if all of them are needed; some contain dynamic libraries
> used by native extensions used in puppet’s depdendencies). Gem.path
> needs to include AIO environment’s gem directory
> (/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0.
>
> The main drawback is that system ruby version needs to be sufficiently
> close to AIO environment one, or some gems will fail to load (I’m
> getting “incompatible library version” error).
Yeah, that would be a recipe for disaster, the compatibility problems it
would introduce would make it fragile.
> Another approach is to rely on puppet API. Currently we require puppet
> gem for our puppet class parser. Starting with puppet api 3.0
> ‘resource_types’ endpoint is available, which allows for retrieval of
> environment’s puppet classes and their parameters. See [2] for some
> examples.
>
> It appears there are some issues with access controls for that API
> point, although they are easily fixed by updating puppet’s auth.conf.
>
>
> I think using API is a much more attractive way of interfacing with
> puppet: we won’t be constrained by ruby vm version, will drop puppet
> dependency, and will eventually get rid of puppet parser code
> (although we’ll need to keep it for a while).
The limitation of the API might be that it only helps with the smart
proxy and not Kafo, which has an identical problem. It might be better
to have a single method or implementation for both projects.
Kafo won't have a master running an API to query (though perhaps the
face would work). Also in Kafo's case we're interested in
documentation, which is more what Puppet Strings provides (#7848).
> Starting with puppet 4.0, one of the the possible installation options
> is to use Puppetlab’s all-in-one (AIO) packages. These come with their
> own run-time environment – ruby vm (currently 2.1.0), gems, native
> extensions, puppet-agent and puppet-server packages. More information
> about the fs layout of AIO packages can be found at [1].
>
>
> Using AIO puppet package with smart-proxy.
>
> Using AIO-packaged puppet is pretty starightforward: $LOAD_PATH needs
> to be prepended with directories in /opt/puppetlabs/puppet/lib/ruby (I
> don’t know if all of them are needed; some contain dynamic libraries
> used by native extensions used in puppet’s depdendencies). Gem.path
> needs to include AIO environment’s gem directory
> (/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0.
>
> The main drawback is that system ruby version needs to be sufficiently
> close to AIO environment one, or some gems will fail to load (I’m
> getting “incompatible library version” error).
>
> Another approach is to rely on puppet API. Currently we require puppet
> gem for our puppet class parser. Starting with puppet api 3.0
> ‘resource_types’ endpoint is available, which allows for retrieval of
> environment’s puppet classes and their parameters. See [2] for some
> examples.
>
> It appears there are some issues with access controls for that API
> point, although they are easily fixed by updating puppet’s auth.conf.
>
>
> I think using API is a much more attractive way of interfacing with
> puppet: we won’t be constrained by ruby vm version, will drop puppet
> dependency, and will eventually get rid of puppet parser code
> (although we’ll need to keep it for a while).
>
> Thoughts?
> -d
I fully agree the API is a much more attractive way to go. In theory you
can decouple the smart-proxy from the puppet server even in terms of
hosts that run them.
IMHO mixing our ruby env with the AIO install is begging for problems.
What Puppet version introduced API 3.0? Is that since Puppet 4.0?
···
On Fri, Nov 06, 2015 at 04:28:37PM +0000, Dmitri Dolguikh wrote:
>> Starting with puppet 4.0, one of the the possible installation options
>> is to use Puppetlab’s all-in-one (AIO) packages. These come with their
>> own run-time environment – ruby vm (currently 2.1.0), gems, native
>> extensions, puppet-agent and puppet-server packages. More information
>> about the fs layout of AIO packages can be found at [1].
>>
>>
>> Using AIO puppet package with smart-proxy.
>>
>> Using AIO-packaged puppet is pretty starightforward: $LOAD_PATH needs
>> to be prepended with directories in /opt/puppetlabs/puppet/lib/ruby (I
>> don’t know if all of them are needed; some contain dynamic libraries
>> used by native extensions used in puppet’s depdendencies). Gem.path
>> needs to include AIO environment’s gem directory
>> (/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0.
>>
>> The main drawback is that system ruby version needs to be sufficiently
>> close to AIO environment one, or some gems will fail to load (I’m
>> getting “incompatible library version” error).
>>
>> Another approach is to rely on puppet API. Currently we require puppet
>> gem for our puppet class parser. Starting with puppet api 3.0
>> ‘resource_types’ endpoint is available, which allows for retrieval of
>> environment’s puppet classes and their parameters. See [2] for some
>> examples.
>>
>> It appears there are some issues with access controls for that API
>> point, although they are easily fixed by updating puppet’s auth.conf.
>>
>>
>> I think using API is a much more attractive way of interfacing with
>> puppet: we won’t be constrained by ruby vm version, will drop puppet
>> dependency, and will eventually get rid of puppet parser code
>> (although we’ll need to keep it for a while).
>>
>> Thoughts?
>> -d
>
> I fully agree the API is a much more attractive way to go. In theory you
> can decouple the smart-proxy from the puppet server even in terms of
> hosts that run them.
>
> IMHO mixing our ruby env with the AIO install is begging for problems.
>
> What Puppet version introduced API 3.0? Is that since Puppet 4.0?
Nods, starting with Puppet 4.0.
-d
···
On Fri, Nov 6, 2015 at 4:40 PM, Ewoud Kohl van Wijngaarden wrote:
> On Fri, Nov 06, 2015 at 04:28:37PM +0000, Dmitri Dolguikh wrote:
>> I think using API is a much more attractive way of interfacing with
>> puppet: we won’t be constrained by ruby vm version, will drop puppet
>> dependency, and will eventually get rid of puppet parser code
>> (although we’ll need to keep it for a while).
+1 for using the API, the other way sounds like Cthulu-in-a-box.
> The limitation of the API might be that it only helps with the smart
> proxy and not Kafo, which has an identical problem. It might be better
> to have a single method or implementation for both projects.
>
> Kafo won't have a master running an API to query (though perhaps the
> face would work). Also in Kafo's case we're interested in
> documentation, which is more what Puppet Strings provides (#7848).
This is probably insane, but in the interest of stimulating discussion…
If the API would be our preferred approach to the proxy, can we invert
the problem for Kafo? That is, could Kafo spin up a local temporary
master configured with an environment of the installer modules in
order to read the classes?
Background to that thought - it's worth noting that from a user
perspective, Kafo gets used once, and the proxy gets used much more
frequently. I think it makes more sense to base design decisions on
the proxy requirements, and see if Kafo can make use of it from there.
Greg
···
On 6 November 2015 at 17:06, Dominic Cleal wrote:
>> Using AIO-packaged puppet is pretty starightforward: $LOAD_PATH needs
>> to be prepended with directories in /opt/puppetlabs/puppet/lib/ruby (I
>> don’t know if all of them are needed; some contain dynamic libraries
>> used by native extensions used in puppet’s depdendencies). Gem.path
>> needs to include AIO environment’s gem directory
>> (/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0.
>>
>> The main drawback is that system ruby version needs to be sufficiently
>> close to AIO environment one, or some gems will fail to load (I’m
>> getting “incompatible library version” error).
>
> Yeah, that would be a recipe for disaster, the compatibility problems it
> would introduce would make it fragile.
>
>> Another approach is to rely on puppet API. Currently we require puppet
>> gem for our puppet class parser. Starting with puppet api 3.0
>> ‘resource_types’ endpoint is available, which allows for retrieval of
>> environment’s puppet classes and their parameters. See [2] for some
>> examples.
>>
>> It appears there are some issues with access controls for that API
>> point, although they are easily fixed by updating puppet’s auth.conf.
>>
>>
>> I think using API is a much more attractive way of interfacing with
>> puppet: we won’t be constrained by ruby vm version, will drop puppet
>> dependency, and will eventually get rid of puppet parser code
>> (although we’ll need to keep it for a while).
>
> The limitation of the API might be that it only helps with the smart
> proxy and not Kafo, which has an identical problem. It might be better
> to have a single method or implementation for both projects.
Nods, API approach will only work for smart-proxy. I think it’s worth
it, however, as the puppet module (or rather a puppet module provider)
is going to be trivial to implement.
>
> Kafo won't have a master running an API to query (though perhaps the
> face would work). Also in Kafo's case we're interested in
> documentation, which is more what Puppet Strings provides (#7848).
>
As an option, we could run kafo inside AIO environment – paths (load
and GEM_HOME/GEM_PATH) are easy to extend, and we’ll know what ruby
version to target.
-d
···
On Fri, Nov 6, 2015 at 5:06 PM, Dominic Cleal wrote:
> On 06/11/15 16:28, Dmitri Dolguikh wrote:
That does sound incredibly complicated. Starting up a master isn't as
easy as it perhaps used to be.
It would also need the API to provide the documentation that Kafo uses,
not just class/parameter lists.
···
On 06/11/15 17:19, Greg Sutcliffe wrote:
> On 6 November 2015 at 17:06, Dominic Cleal wrote:
>> The limitation of the API might be that it only helps with the smart
>> proxy and not Kafo, which has an identical problem. It might be better
>> to have a single method or implementation for both projects.
>>
>> Kafo won't have a master running an API to query (though perhaps the
>> face would work). Also in Kafo's case we're interested in
>> documentation, which is more what Puppet Strings provides (#7848).
>
> This is probably insane, but in the interest of stimulating discussion...
>
> If the API would be our preferred approach to the proxy, can we invert
> the problem for Kafo? That is, could Kafo spin up a local temporary
> master configured with an environment of the installer modules in
> order to read the classes?
One option is to have a common solution for both Kafo and smart-proxy:
it will be installed and executed inside of AIO environment, and
expose API to support puppet class parsing.
My preference would be to use puppet http api for smart-proxy (minimal
dependencies, works for any type of Puppet 4.x environment, minimal
amount of code to maintain). Perhaps Kafo devs could chime in with
their preferences re: interfacing with AIO-packaged puppet? If this
would help, I could look into executing Kafo inside AIO.
Cheers,
-d
···
On Fri, Nov 6, 2015 at 5:51 PM, Dmitri Dolguikh wrote:
> On Fri, Nov 6, 2015 at 5:06 PM, Dominic Cleal wrote:
>> On 06/11/15 16:28, Dmitri Dolguikh wrote:
>>> Using AIO-packaged puppet is pretty starightforward: $LOAD_PATH needs
>>> to be prepended with directories in /opt/puppetlabs/puppet/lib/ruby (I
>>> don’t know if all of them are needed; some contain dynamic libraries
>>> used by native extensions used in puppet’s depdendencies). Gem.path
>>> needs to include AIO environment’s gem directory
>>> (/opt/puppetlabs/puppet/lib/ruby/gems/2.1.0.
>>>
>>> The main drawback is that system ruby version needs to be sufficiently
>>> close to AIO environment one, or some gems will fail to load (I’m
>>> getting “incompatible library version” error).
>>
>> Yeah, that would be a recipe for disaster, the compatibility problems it
>> would introduce would make it fragile.
>>
>>> Another approach is to rely on puppet API. Currently we require puppet
>>> gem for our puppet class parser. Starting with puppet api 3.0
>>> ‘resource_types’ endpoint is available, which allows for retrieval of
>>> environment’s puppet classes and their parameters. See [2] for some
>>> examples.
>>>
>>> It appears there are some issues with access controls for that API
>>> point, although they are easily fixed by updating puppet’s auth.conf.
>>>
>>>
>>> I think using API is a much more attractive way of interfacing with
>>> puppet: we won’t be constrained by ruby vm version, will drop puppet
>>> dependency, and will eventually get rid of puppet parser code
>>> (although we’ll need to keep it for a while).
>>
>> The limitation of the API might be that it only helps with the smart
>> proxy and not Kafo, which has an identical problem. It might be better
>> to have a single method or implementation for both projects.
>
> Nods, API approach will only work for smart-proxy. I think it’s worth
> it, however, as the puppet module (or rather a puppet module provider)
> is going to be trivial to implement.
>
>>
>> Kafo won't have a master running an API to query (though perhaps the
>> face would work). Also in Kafo's case we're interested in
>> documentation, which is more what Puppet Strings provides (#7848).
>>
>
> As an option, we could run kafo inside AIO environment -- paths (load
> and GEM_HOME/GEM_PATH) are easy to extend, and we’ll know what ruby
> version to target.
>
> -d
>
>> --
>> Dominic Cleal
>> dominic@cleal.org
>>
>> --
>> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
> To summarize:
>
> One option is to have a common solution for both Kafo and smart-proxy:
> it will be installed and executed inside of AIO environment, and
> expose API to support puppet class parsing.
>
> Another option is to deal with Kafo and smart-proxy needs separately.
> Smart-proxy can use puppet ‘resource_types’ http api (see
> https://docs.puppetlabs.com/puppet/4.2/reference/http_api/http_resource_type.html#search-with-results).
> We cannot rely on master being present during Kafo execution, and
> therefore cannot rely on the above-mentioned api. We could: add a
> piece of code executing inside AIO that exposes API for class parsing,
> or execute Kafo inside AIO environment.
> I don’t know the history behind
> Feature #11992: Support loading Puppet from AIO packages - Smart Proxy - Foreman, is installing a puppet
> gem in parallel to AIO an option?
You're back to a few problems then, a) reduced Ruby version support in
Puppet limits what the smart proxy can run on, b) smart proxy packages
would need to ship this Puppet gem which I don't think is our business,
c) Puppet versions may not match and cause different results.
> My preference would be to use puppet http api for smart-proxy (minimal
> dependencies, works for any type of Puppet 4.x environment, minimal
> amount of code to maintain). Perhaps Kafo devs could chime in with
> their preferences re: interfacing with AIO-packaged puppet? If this
> would help, I could look into executing Kafo inside AIO.
If you like, but I think you'd have a lot of problems trying to package
that, especially while retaining compatibility with what's currently there.
>> To summarize:
>>
>> One option is to have a common solution for both Kafo and smart-proxy:
>> it will be installed and executed inside of AIO environment, and
>> expose API to support puppet class parsing.
>>
>> Another option is to deal with Kafo and smart-proxy needs separately.
>> Smart-proxy can use puppet ‘resource_types’ http api (see
>> https://docs.puppetlabs.com/puppet/4.2/reference/http_api/http_resource_type.html#search-with-results).
>> We cannot rely on master being present during Kafo execution, and
>> therefore cannot rely on the above-mentioned api. We could: add a
>> piece of code executing inside AIO that exposes API for class parsing,
>> or execute Kafo inside AIO environment.
>> I don’t know the history behind
>> Feature #11992: Support loading Puppet from AIO packages - Smart Proxy - Foreman, is installing a puppet
>> gem in parallel to AIO an option?
>
> You're back to a few problems then, a) reduced Ruby version support in
> Puppet limits what the smart proxy can run on, b) smart proxy packages
> would need to ship this Puppet gem which I don't think is our business,
> c) Puppet versions may not match and cause different results.
Apologies, I should’ve been more clear: my question was in relation to
Kafo, I think using puppet api in smart-proxy is a better way forward.
Points b and c are still applicable though.
-d
···
On Tue, Nov 10, 2015 at 11:23 AM, Dominic Cleal wrote:
> On 10/11/15 10:46, Dmitri Dolguikh wrote:
My preference would be to use puppet http api for smart-proxy (minimal
dependencies, works for any type of Puppet 4.x environment, minimal
amount of code to maintain). Perhaps Kafo devs could chime in with
their preferences re: interfacing with AIO-packaged puppet? If this
would help, I could look into executing Kafo inside AIO.
If you like, but I think you’d have a lot of problems trying to package
that, especially while retaining compatibility with what’s currently there.
