Making the proxy extensible

At the community meeting last week we talked a little about making the smart-proxy "pluggable" - namely able to be extended by gems. One example use case is adding DNS management by a third party, like Route 53 or Dynect. Another reason this would be nice is to make the proxy even smaller; there's no reason for something like a DHCP or DNS server to have to include all the Puppet, CA, TFTP, and other code; this is more a secondary benefit than a primary motivation, IMO.

I started implementing a design a couple months ago, but have unfortunately been wrapped up working on other stuff since then. The idea behind the refactoring I started was two-fold:

  1. Turn the proxy into an autoloader that exposes API endpoints that are stored in gems.
  2. Move some stuff to be an "internal gem" (basically just library code that's well abstracted) and other things get completely removed from the proxy into gems on rubygems.org with names like foreman-proxy-dhcp.

So the first part is relatively straight forward, but the second item requires some more thought. What do you all think should be included in the proxy by default? Here's my list:

  • Puppet
  • Puppet CA
  • TFTP

The reason I chose those three is that they are what's needed to setup a basic "all-in-one" system.

So that's what's on my mind with making the proxy pluggable - a little vague, but that's why I'm sending this email :slight_smile:

Thoughts?

-Sam

>
> At the community meeting last week we talked a little about making the
smart-proxy "pluggable" - namely able to be extended by gems. One example
use case is adding DNS management by a third party, like Route 53 or
Dynect. Another reason this would be nice is to make the proxy even
smaller; there's no reason for something like a DHCP or DNS server to have
to include all the Puppet, CA, TFTP, and other code; this is more a
secondary benefit than a primary motivation, IMO.
>
> I started implementing a design a couple months ago, but have
unfortunately been wrapped up working on other stuff since then. The idea
behind the refactoring I started was two-fold:
>
> 1. Turn the proxy into an autoloader that exposes API endpoints that are
stored in gems.
> 2. Move some stuff to be an "internal gem" (basically just library code
that's well abstracted) and other things get completely removed from the
proxy into gems on rubygems.org with names like foreman-proxy-dhcp.

Were you using rack for this? If not, thoughts on using rack?

-d
>
> So the first part is relatively straight forward, but the second item
requires some more thought. What do you all think should be included in the
proxy by default? Here's my list:
>
> * Puppet
> * Puppet CA
> * TFTP
>
> The reason I chose those three is that they are what's needed to setup a
basic "all-in-one" system.
>
> So that's what's on my mind with making the proxy pluggable - a little
vague, but that's why I'm sending this email :slight_smile:
>
> Thoughts?
>
> -Sam
>
> –
> You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.

··· On 6 May 2013 04:49, "Sam Kottler" wrote: > For more options, visit https://groups.google.com/groups/opt_out. > >

> From: "Dmitri Dolguikh" <witlessbird@gmail.com>
> To: foreman-dev@googlegroups.com
> Sent: Monday, May 6, 2013 3:31:00 AM
> Subject: Re: [foreman-dev] Making the proxy extensible
>
> >
> > At the community meeting last week we talked a little about making the
> smart-proxy "pluggable" - namely able to be extended by gems. One example
> use case is adding DNS management by a third party, like Route 53 or
> Dynect. Another reason this would be nice is to make the proxy even
> smaller; there's no reason for something like a DHCP or DNS server to have
> to include all the Puppet, CA, TFTP, and other code; this is more a
> secondary benefit than a primary motivation, IMO.
> >
> > I started implementing a design a couple months ago, but have
> unfortunately been wrapped up working on other stuff since then. The idea
> behind the refactoring I started was two-fold:
> >
> > 1. Turn the proxy into an autoloader that exposes API endpoints that are
> stored in gems.
> > 2. Move some stuff to be an "internal gem" (basically just library code
> that's well abstracted) and other things get completely removed from the
> proxy into gems on rubygems.org with names like foreman-proxy-dhcp.
>
> Were you using rack for this? If not, thoughts on using rack?

You mean writing each piece of the proxy as rack middleware? I hadn't thought much about that, but it's an interesting idea.

I've only written a trivial piece of rack middleware before - have you written anything more substantive? If so, how does it scale in terms of remaining maintainable as the number and size of extensions grow?

··· ----- Original Message ----- > On 6 May 2013 04:49, "Sam Kottler" wrote:

-d

So the first part is relatively straight forward, but the second item
requires some more thought. What do you all think should be included in the
proxy by default? Here’s my list:

  • Puppet
  • Puppet CA
  • TFTP

The reason I chose those three is that they are what’s needed to setup a
basic “all-in-one” system.

So that’s what’s on my mind with making the proxy pluggable - a little
vague, but that’s why I’m sending this email :slight_smile:

Thoughts?

-Sam

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

>
>
> > From: "Dmitri Dolguikh" <witlessbird@gmail.com>
> > To: foreman-dev@googlegroups.com
> > Sent: Monday, May 6, 2013 3:31:00 AM
> > Subject: Re: [foreman-dev] Making the proxy extensible
> >
> > >
> > > At the community meeting last week we talked a little about making the
> > smart-proxy "pluggable" - namely able to be extended by gems. One example
> > use case is adding DNS management by a third party, like Route 53 or
> > Dynect. Another reason this would be nice is to make the proxy even
> > smaller; there's no reason for something like a DHCP or DNS server to
> have
> > to include all the Puppet, CA, TFTP, and other code; this is more a
> > secondary benefit than a primary motivation, IMO.
> > >
> > > I started implementing a design a couple months ago, but have
> > unfortunately been wrapped up working on other stuff since then. The idea
> > behind the refactoring I started was two-fold:
> > >
> > > 1. Turn the proxy into an autoloader that exposes API endpoints that
> are
> > stored in gems.
> > > 2. Move some stuff to be an "internal gem" (basically just library code
> > that's well abstracted) and other things get completely removed from the
> > proxy into gems on rubygems.org with names like foreman-proxy-dhcp.
> >
> > Were you using rack for this? If not, thoughts on using rack?
>
> You mean writing each piece of the proxy as rack middleware? I hadn't
> thought much about that, but it's an interesting idea.
>

I wasn't originally thinking about rack middleware, but with
https://github.com/rack/rack-contrib/blob/master/lib/rack/contrib/simple_endpoint.rb,
we could (or we could use custom routing).

>
> I've only written a trivial piece of rack middleware before - have you
> written anything more substantive? If so, how does it scale in terms of
> remaining maintainable as the number and size of extensions grow?
>
>
I don't have any experience writing middleware, but seeing that rails is
pretty much bunch of middleware + bunch of railties I don't expect much
trouble. I think middleware might work out nice, as it imposes very little
on the middleware (and we'd be free to define modules' architecture and
layout)…

-d

··· On Mon, May 6, 2013 at 10:12 PM, Sam Kottler wrote: > ----- Original Message ----- > > On 6 May 2013 04:49, "Sam Kottler" wrote:

-d

So the first part is relatively straight forward, but the second item
requires some more thought. What do you all think should be included in
the
proxy by default? Here’s my list:

  • Puppet
  • Puppet CA
  • TFTP

The reason I chose those three is that they are what’s needed to setup
a
basic “all-in-one” system.

So that’s what’s on my mind with making the proxy pluggable - a little
vague, but that’s why I’m sending this email :slight_smile:

Thoughts?

-Sam

–
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.

To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

sinatra is not a big overhead and brings a lot to the table that we
wouldn't have it re-implement ourselves. That and the proxy is currently
written using sinatra, is should be easy to extract the current components
into plugable modules.

Light travels faster than sound. This is why some people appear bright
until you hear them speak

··· On Tue, May 7, 2013 at 12:26 PM, Dmitri Dolguikh wrote:

On Mon, May 6, 2013 at 10:12 PM, Sam Kottler skottler@redhat.com wrote:

----- Original Message -----

From: “Dmitri Dolguikh” witlessbird@gmail.com
To: foreman-dev@googlegroups.com
Sent: Monday, May 6, 2013 3:31:00 AM
Subject: Re: [foreman-dev] Making the proxy extensible

On 6 May 2013 04:49, “Sam Kottler” skottler@redhat.com wrote:

At the community meeting last week we talked a little about making the
smart-proxy “pluggable” - namely able to be extended by gems. One
example
use case is adding DNS management by a third party, like Route 53 or
Dynect. Another reason this would be nice is to make the proxy even
smaller; there’s no reason for something like a DHCP or DNS server to
have
to include all the Puppet, CA, TFTP, and other code; this is more a
secondary benefit than a primary motivation, IMO.

I started implementing a design a couple months ago, but have
unfortunately been wrapped up working on other stuff since then. The
idea
behind the refactoring I started was two-fold:

  1. Turn the proxy into an autoloader that exposes API endpoints that
    are
    stored in gems.
  1. Move some stuff to be an “internal gem” (basically just library
    code
    that’s well abstracted) and other things get completely removed from the
    proxy into gems on rubygems.org with names like foreman-proxy-dhcp.

Were you using rack for this? If not, thoughts on using rack?

You mean writing each piece of the proxy as rack middleware? I hadn’t
thought much about that, but it’s an interesting idea.

I wasn’t originally thinking about rack middleware, but with
https://github.com/rack/rack-contrib/blob/master/lib/rack/contrib/simple_endpoint.rb,
we could (or we could use custom routing).

I’ve only written a trivial piece of rack middleware before - have you
written anything more substantive? If so, how does it scale in terms of
remaining maintainable as the number and size of extensions grow?

I don’t have any experience writing middleware, but seeing that rails is
pretty much bunch of middleware + bunch of railties I don’t expect much
trouble. I think middleware might work out nice, as it imposes very little
on the middleware (and we’d be free to define modules’ architecture and
layout)…

-d

-d

So the first part is relatively straight forward, but the second item
requires some more thought. What do you all think should be included in
the
proxy by default? Here’s my list:

  • Puppet
  • Puppet CA
  • TFTP

The reason I chose those three is that they are what’s needed to
setup a
basic “all-in-one” system.

So that’s what’s on my mind with making the proxy pluggable - a little
vague, but that’s why I’m sending this email :slight_smile:

Thoughts?

-Sam

–
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.

To unsubscribe from this group and stop receiving emails from it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

> sinatra is not a big overhead and brings a lot to the table that we
> wouldn't have it re-implement ourselves. That and the proxy is currently
> written using sinatra, is should be easy to extract the current components
> into plugable modules.
>
>
We could make individual components (sub-proxies?) sinatra apps, and then
mount them all together using rack-mount? This way each module has its own
routes, etc…
-d

··· On Tue, May 7, 2013 at 5:57 PM, Telmo X wrote:

Light travels faster than sound. This is why some people appear bright
until you hear them speak

On Tue, May 7, 2013 at 12:26 PM, Dmitri Dolguikh witlessbird@gmail.comwrote:

On Mon, May 6, 2013 at 10:12 PM, Sam Kottler skottler@redhat.com wrote:

----- Original Message -----

From: “Dmitri Dolguikh” witlessbird@gmail.com
To: foreman-dev@googlegroups.com
Sent: Monday, May 6, 2013 3:31:00 AM
Subject: Re: [foreman-dev] Making the proxy extensible

On 6 May 2013 04:49, “Sam Kottler” skottler@redhat.com wrote:

At the community meeting last week we talked a little about making
the
smart-proxy “pluggable” - namely able to be extended by gems. One
example
use case is adding DNS management by a third party, like Route 53 or
Dynect. Another reason this would be nice is to make the proxy even
smaller; there’s no reason for something like a DHCP or DNS server to
have
to include all the Puppet, CA, TFTP, and other code; this is more a
secondary benefit than a primary motivation, IMO.

I started implementing a design a couple months ago, but have
unfortunately been wrapped up working on other stuff since then. The
idea
behind the refactoring I started was two-fold:

  1. Turn the proxy into an autoloader that exposes API endpoints that
    are
    stored in gems.
  1. Move some stuff to be an “internal gem” (basically just library
    code
    that’s well abstracted) and other things get completely removed from
    the
    proxy into gems on rubygems.org with names like foreman-proxy-dhcp.

Were you using rack for this? If not, thoughts on using rack?

You mean writing each piece of the proxy as rack middleware? I hadn’t
thought much about that, but it’s an interesting idea.

I wasn’t originally thinking about rack middleware, but with
https://github.com/rack/rack-contrib/blob/master/lib/rack/contrib/simple_endpoint.rb,
we could (or we could use custom routing).

I’ve only written a trivial piece of rack middleware before - have you
written anything more substantive? If so, how does it scale in terms of
remaining maintainable as the number and size of extensions grow?

I don’t have any experience writing middleware, but seeing that rails is
pretty much bunch of middleware + bunch of railties I don’t expect much
trouble. I think middleware might work out nice, as it imposes very little
on the middleware (and we’d be free to define modules’ architecture and
layout)…

-d

-d

So the first part is relatively straight forward, but the second item
requires some more thought. What do you all think should be included
in the
proxy by default? Here’s my list:

  • Puppet
  • Puppet CA
  • TFTP

The reason I chose those three is that they are what’s needed to
setup a
basic “all-in-one” system.

So that’s what’s on my mind with making the proxy pluggable - a
little
vague, but that’s why I’m sending this email :slight_smile:

Thoughts?

-Sam

–
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.

To unsubscribe from this group and stop receiving emails from it,
send an
email to foreman-dev+unsubscribe@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google
Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send
an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I know it’s a 6yr old thread. I do see that the Route53 smart-proxy is available at this point. Has anyone ever been able to plug a Dynect DNS provider to smart-proxy?

There’s a wiki page that describes how to do so at How to Create a Smart-Proxy Plugin - Foreman and a template:

As long as there’s an API, it’s usually very easy. It’d be best to open a new thread.

1 Like