[RFC] - Proxies with multiple interfaces

Development
1

Hey!

So lzap suggested this… lets see how it goes :slight_smile:

I'm looking to solve the following issues

  • I would like to have a multi-homed proxy.
  • I would like clients to connect to a Proxy when there is a NAT between.

My approach has been for proxies to have multiple URLs and during
provisioning the user would then select a URL instead of the current method
of selecting a Proxy, This would be then mean all communication between the
Client -> Proxy would be via the selected URL. Each proxy would also have a
'primary' URL which Foreman would always use to connect to it via.

Does this approach make sense?
Any feedback to greatly appreciated

Changes required:
Foreman: https://github.com/theforeman/foreman/pull/4346

  • Adds Proxy URL concept
  • Puppet CA & Puppet Master selection are now URLS

Katello: PR to be raised
Content Source selection need to list URLS.
Bootstrap RPM will need to create one for each URL

Let me know what you think or any questions you have!
Sean

2

Hey,

I can't remember recommending such a big change :slight_smile: Can you elaborate
more on motivation? Multi-homed proxy is still one instance, why would
Foreman need to reach to it in two different ways? Isn't the result
always the same (a change on the same instance)? How about relaxing
the constraing when each individual proxy must have unique URL? Just
checking before introducing another table (more SQL joins and query
complexity).

By the way this is (a properly tagged) RFC thread, why you already
imlemented that? You had to spend lot of time on this already, we may
find an alternative and less intrusive solution instead.

LZ

··· On Wed, Mar 15, 2017 at 11:18 AM, Sean O'Keeffe wrote: > Hey! > > So lzap suggested this.. lets see how it goes :-) > > I'm looking to solve the following issues > - I would like to have a multi-homed proxy. > - I would like clients to connect to a Proxy when there is a NAT between. > > My approach has been for proxies to have multiple URLs and during > provisioning the user would then select a URL instead of the current method > of selecting a Proxy, This would be then mean all communication between the > Client -> Proxy would be via the selected URL. Each proxy would also have a > 'primary' URL which Foreman would always use to connect to it via. > > Does this approach make sense? > Any feedback to greatly appreciated > > Changes required: > Foreman: https://github.com/theforeman/foreman/pull/4346 > - Adds Proxy URL concept > - Puppet CA & Puppet Master selection are now URLS > > Katello: PR to be raised > Content Source selection need to list URLS. > Bootstrap RPM will need to create one for each URL > > Let me know what you think or any questions you have! > Sean > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas @lzap Zapletal

3

I have an alternative, although I am not sure it's better:
We can introduce "reachability" concept to our subnets. Something that
would say "You can get from subnet X to subnet Y".
Once this information is available to foreman, we can assign proxies (and
foreman itself) to a subnet.
When a host requests a proxy, the subnets would be compared, and used if
they are reachable. The same will go for Foreman -> proxy interactions,
like in REX.

Advantages: Adds more knowledge of the real world into foreman, a fact that
could be used later. Does not introduce new objects, the users are already
familiar with subnets concept.
Disadvantages: More input from the user required. The same proxy would be
registered more than once - for each subnet it is reachable from.

If we go with the original solution, I would suggest duplicating records in
the current table, but adding a field that would indicate which proxies are
duplicates.
The UI then would be able to group the records by that additional field,
and show a single object for each group.

Advantages: Amount of joins stays the same, the UI for simple cases stays
the same.
Disadvantages: Less intuitive implementation.

··· On Wednesday, March 15, 2017 at 12:18:50 PM UTC+2, Sean O'Keeffe wrote: > > Hey! > > So lzap suggested this.. lets see how it goes :-) > > *I'm looking to solve the following issues* > - I would like to have a multi-homed proxy. > - I would like clients to connect to a Proxy when there is a NAT between. > > My approach has been for proxies to have multiple URLs and during > provisioning the user would then select a URL instead of the current method > of selecting a Proxy, This would be then mean all communication between the > Client -> Proxy would be via the selected URL. Each proxy would also have a > 'primary' URL which Foreman would always use to connect to it via. > > Does this approach make sense? > Any feedback to greatly appreciated > > *Changes required:* > Foreman: https://github.com/theforeman/foreman/pull/4346 > - Adds Proxy URL concept > - Puppet CA & Puppet Master selection are now URLS > > Katello: PR to be raised > Content Source selection need to list URLS. > Bootstrap RPM will need to create one for each URL > > Let me know what you think or any questions you have! > Sean >
4

> Hey,
>
Howdy :slight_smile:

>
> I can't remember recommending such a big change :slight_smile:

Sorry I meant about tagging the email with [RFE]

> Can you elaborate more on motivation? Multi-homed proxy is still one
> instance, why would
> Foreman need to reach to it in two different ways?

This is for clients to reach a proxy in the different ways, NOT foreman.

> Isn't the result always the same (a change on the same instance)? How
> about relaxing
> the constraing when each individual proxy must have unique URL? Just
> checking before introducing another table (more SQL joins and query
> complexity).
>
If a Host / HostGroups can be assigned a URL/Hostname instead of a proxy
its useful for Multi-homing, loadbalancing & NAT'ed environments, maybe
others?
What does relaxing the unique URL constrain actually give us? I think if
you have 2 proxies with the same URL that's just effectively duplicating
the object?

Or if you mean creating 2 proxies with different URLs (an external & and
another internal one) I think would be problematic when Foreman needs to
talk to the Proxy. E.g Host needs to talk to proxy1-external.example.com
but foreman must talk to the proxy via proxy1-internal.example.com. If we
have 2 proxies in foreman ( proxy1-internal & proxy1-external ) and you
assign a host proxy1-external foreman would try to connect to
proxy1-external.example.com to create a DHCP or DNS or something else with
would not work.

>
> By the way this is (a properly tagged) RFC thread, why you already
> imlemented that? You had to spend lot of time on this already, we may
> find an alternative and less intrusive solution instead.
>
Because IMO proper discussions don't tend to actually happen until someone
writes the code.

Thanks for the reply!
Sean

··· On Mon, Mar 20, 2017 at 2:25 PM, Lukas Zapletal wrote:

LZ

On Wed, Mar 15, 2017 at 11:18 AM, Sean O’Keeffe sokeeffe@redhat.com > wrote:

Hey!

So lzap suggested this… lets see how it goes :slight_smile:

I’m looking to solve the following issues

  • I would like to have a multi-homed proxy.
  • I would like clients to connect to a Proxy when there is a NAT between.

My approach has been for proxies to have multiple URLs and during
provisioning the user would then select a URL instead of the current
method
of selecting a Proxy, This would be then mean all communication between
the
Client -> Proxy would be via the selected URL. Each proxy would also
have a
’primary’ URL which Foreman would always use to connect to it via.

Does this approach make sense?
Any feedback to greatly appreciated

Changes required:
Foreman: https://github.com/theforeman/foreman/pull/4346

  • Adds Proxy URL concept
  • Puppet CA & Puppet Master selection are now URLS

Katello: PR to be raised
Content Source selection need to list URLS.
Bootstrap RPM will need to create one for each URL

Let me know what you think or any questions you have!
Sean

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas @lzap Zapletal

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

That sounds very much like you are suggesting that Foreman maintains
routing tables. Manually or automatically? Either will escalate very
quickly.
-d

··· On Tue, Apr 4, 2017 at 7:18 PM, wrote: > > I have an alternative, although I am not sure it's better: > We can introduce "reachability" concept to our subnets. Something that would > say "You can get from subnet X to subnet Y".
6

> > Can you elaborate more on motivation? Multi-homed proxy is still one
> > instance, why would
> > Foreman need to reach to it in two different ways?
>
> This is for clients to reach a proxy in the different ways, NOT foreman.
>
>
> > Isn't the result always the same (a change on the same instance)? How
> > about relaxing
> > the constraing when each individual proxy must have unique URL? Just
> > checking before introducing another table (more SQL joins and query
> > complexity).
> >
> If a Host / HostGroups can be assigned a URL/Hostname instead of a proxy
> its useful for Multi-homing, loadbalancing & NAT'ed environments, maybe
> others?
> What does relaxing the unique URL constrain actually give us? I think if
> you have 2 proxies with the same URL that's just effectively duplicating
> the object?
>
> Or if you mean creating 2 proxies with different URLs (an external & and
> another internal one) I think would be problematic when Foreman needs to
> talk to the Proxy. E.g Host needs to talk to proxy1-external.example.com
> but foreman must talk to the proxy via proxy1-internal.example.com. If we
> have 2 proxies in foreman ( proxy1-internal & proxy1-external ) and you
> assign a host proxy1-external foreman would try to connect to
> proxy1-external.example.com to create a DHCP or DNS or something else with
> would not work.

I think a different URL for clients and Foreman itself makes a lot of
sense. Now I have some trouble reading the design from the code. Could
you describe how this would be used by a user? Maybe a screenshot from
the new configuration page.

··· On Mon, Mar 20, 2017 at 03:50:30PM +0000, Sean O'Keeffe wrote: > On Mon, Mar 20, 2017 at 2:25 PM, Lukas Zapletal wrote:

On Wed, Mar 15, 2017 at 11:18 AM, Sean O’Keeffe sokeeffe@redhat.com > > wrote:

Hey!

So lzap suggested this… lets see how it goes :slight_smile:

I’m looking to solve the following issues

  • I would like to have a multi-homed proxy.
  • I would like clients to connect to a Proxy when there is a NAT between.

My approach has been for proxies to have multiple URLs and during
provisioning the user would then select a URL instead of the current
method
of selecting a Proxy, This would be then mean all communication between
the
Client -> Proxy would be via the selected URL. Each proxy would also
have a
’primary’ URL which Foreman would always use to connect to it via.

Does this approach make sense?
Any feedback to greatly appreciated

Changes required:
Foreman: https://github.com/theforeman/foreman/pull/4346

  • Adds Proxy URL concept
  • Puppet CA & Puppet Master selection are now URLS

Katello: PR to be raised
Content Source selection need to list URLS.
Bootstrap RPM will need to create one for each URL

Let me know what you think or any questions you have!
Sean

7

> > > Can you elaborate more on motivation? Multi-homed proxy is still one
> > > instance, why would
> > > Foreman need to reach to it in two different ways?
> >
> > This is for clients to reach a proxy in the different ways, NOT foreman.
> >
> >
> > > Isn't the result always the same (a change on the same instance)? How
> > > about relaxing
> > > the constraing when each individual proxy must have unique URL? Just
> > > checking before introducing another table (more SQL joins and query
> > > complexity).
> > >
> > If a Host / HostGroups can be assigned a URL/Hostname instead of a proxy
> > its useful for Multi-homing, loadbalancing & NAT'ed environments, maybe
> > others?
> > What does relaxing the unique URL constrain actually give us? I think if
> > you have 2 proxies with the same URL that's just effectively duplicating
> > the object?
> >
> > Or if you mean creating 2 proxies with different URLs (an external & and
> > another internal one) I think would be problematic when Foreman needs to
> > talk to the Proxy. E.g Host needs to talk to proxy1-external.example.com
> > but foreman must talk to the proxy via proxy1-internal.example.com. If
> we
> > have 2 proxies in foreman ( proxy1-internal & proxy1-external ) and you
> > assign a host proxy1-external foreman would try to connect to
> > proxy1-external.example.com to create a DHCP or DNS or something else
> with
> > would not work.
>
> I think a different URL for clients and Foreman itself makes a lot of
> sense. Now I have some trouble reading the design from the code. Could
> you describe how this would be used by a user? Maybe a screenshot from
> the new configuration page.
>

Sure,

I've added another tab to the Smart Proxies form, where you can add
"Secondary URLs". On the Host/Hostgroup from the user can then select a URL
for attributes like Puppet Master, Puppet CA (and Openscap and content
source for those plugins). I've uploaded some pictures to the PR as well
which should explain it much better.

https://github.com/theforeman/foreman/pull/4346#issuecomment-288461410

One thing Stephen mentioned on the PR was if URLs could have many proxies
as well as proxies have many URLs. That would go someway to better load
balanced proxies. Which I think would be great, though I personally don't
like the idea of another form & menu item just to create URLs, I think we
have lots of forms and menu items already…

Sean

··· On Wed, Mar 22, 2017 at 1:40 PM, Ewoud Kohl van Wijngaarden < ewoud@kohlvanwijngaarden.nl> wrote: > On Mon, Mar 20, 2017 at 03:50:30PM +0000, Sean O'Keeffe wrote: > > On Mon, Mar 20, 2017 at 2:25 PM, Lukas Zapletal wrote:

On Wed, Mar 15, 2017 at 11:18 AM, Sean O’Keeffe sokeeffe@redhat.com > > > wrote:

Hey!

So lzap suggested this… lets see how it goes :slight_smile:

I’m looking to solve the following issues

  • I would like to have a multi-homed proxy.
  • I would like clients to connect to a Proxy when there is a NAT
    between.

My approach has been for proxies to have multiple URLs and during
provisioning the user would then select a URL instead of the current
method
of selecting a Proxy, This would be then mean all communication
between

the

Client -> Proxy would be via the selected URL. Each proxy would also
have a
‘primary’ URL which Foreman would always use to connect to it via.

Does this approach make sense?
Any feedback to greatly appreciated

Changes required:
Foreman: https://github.com/theforeman/foreman/pull/4346

  • Adds Proxy URL concept
  • Puppet CA & Puppet Master selection are now URLS

Katello: PR to be raised
Content Source selection need to list URLS.
Bootstrap RPM will need to create one for each URL

Let me know what you think or any questions you have!
Sean

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

>
> > > > Can you elaborate more on motivation? Multi-homed proxy is still one
> > > > instance, why would
> > > > Foreman need to reach to it in two different ways?
> > >
> > > This is for clients to reach a proxy in the different ways, NOT foreman.
> > >
> > >
> > > > Isn't the result always the same (a change on the same instance)? How
> > > > about relaxing
> > > > the constraing when each individual proxy must have unique URL? Just
> > > > checking before introducing another table (more SQL joins and query
> > > > complexity).
> > > >
> > > If a Host / HostGroups can be assigned a URL/Hostname instead of a proxy
> > > its useful for Multi-homing, loadbalancing & NAT'ed environments, maybe
> > > others?
> > > What does relaxing the unique URL constrain actually give us? I think if
> > > you have 2 proxies with the same URL that's just effectively duplicating
> > > the object?
> > >
> > > Or if you mean creating 2 proxies with different URLs (an external & and
> > > another internal one) I think would be problematic when Foreman needs to
> > > talk to the Proxy. E.g Host needs to talk to proxy1-external.example.com
> > > but foreman must talk to the proxy via proxy1-internal.example.com. If
> > we
> > > have 2 proxies in foreman ( proxy1-internal & proxy1-external ) and you
> > > assign a host proxy1-external foreman would try to connect to
> > > proxy1-external.example.com to create a DHCP or DNS or something else
> > with
> > > would not work.
> >
> > I think a different URL for clients and Foreman itself makes a lot of
> > sense. Now I have some trouble reading the design from the code. Could
> > you describe how this would be used by a user? Maybe a screenshot from
> > the new configuration page.
> >
>
> Sure,
>
> I've added another tab to the Smart Proxies form, where you can add
> "Secondary URLs". On the Host/Hostgroup from the user can then select a URL
> for attributes like Puppet Master, Puppet CA (and Openscap and content
> source for those plugins). I've uploaded some pictures to the PR as well
> which should explain it much better.
>
> https://github.com/theforeman/foreman/pull/4346#issuecomment-288461410
>
> One thing Stephen mentioned on the PR was if URLs could have many proxies
> as well as proxies have many URLs. That would go someway to better load
> balanced proxies. Which I think would be great, though I personally don't
> like the idea of another form & menu item just to create URLs, I think we
> have lots of forms and menu items already…

As a user it feels very complex and overwhelming. This is a perfect
example where a use case for a complex deployment makes it harder for
simple deployments.

I'm not against it, but just some thinking out loud to see if we both
get what the impact is.

How often do you have multiple URLs? My feeling is that there are the
following use cases:

A proxy has 1 URL

This is the current situation. Speaks for itself. Question here is how
you display a proxy with only 1 URL. Do you simplify it so it matches
the current UI or keep it consistent with multiple URLs?

A proxy has 2 URLs

Here we have a separate network for Foreman and its proxies. Then the
clients talk to their proxies on another network for other services
(puppet, DHCP etc).
That means we have a URL for the internal network and a URL for the
client network.

A proxy has multiple URLs for multiple clients

A proxy could live in multiple client networks and serve them with
different names. I'm wondering how common this is and it's really an
added benefit. If you're separating anyway, maybe we should just
recommend deploying multiple proxies.

Regarding the use case where a URL has multiple proxies: this is a proxy
cluster. You could reason the other way around. Let me explain that.
Currently a proxy has multiple features. Each feature is reachable under
the same URL as a proxy. What you could consider is that you have the
proxy feature class.

ProxyFeature

  • name (puppet, tftp)
  • URL
  • foreman proxies (1 or more)

Now you do need some logic to know if Foreman needs to talk to 1 of the
proxies or all. In the case of importing puppet classes you just need 1,
but in case of tftp all need to download. Some features might not allow
multiple proxies.

One complexity might be that not everything is a URL. tftp is just an IP
or hostname.

Does that make sense?

··· On Wed, Mar 22, 2017 at 04:44:51PM +0000, Sean O'Keeffe wrote: > On Wed, Mar 22, 2017 at 1:40 PM, Ewoud Kohl van Wijngaarden < > ewoud@kohlvanwijngaarden.nl> wrote: > > On Mon, Mar 20, 2017 at 03:50:30PM +0000, Sean O'Keeffe wrote: > > > On Mon, Mar 20, 2017 at 2:25 PM, Lukas Zapletal wrote:

On Wed, Mar 15, 2017 at 11:18 AM, Sean O’Keeffe sokeeffe@redhat.com > > > > wrote:

Hey!

So lzap suggested this… lets see how it goes :slight_smile:

I’m looking to solve the following issues

  • I would like to have a multi-homed proxy.
  • I would like clients to connect to a Proxy when there is a NAT
    between.

My approach has been for proxies to have multiple URLs and during
provisioning the user would then select a URL instead of the current
method
of selecting a Proxy, This would be then mean all communication
between

the

Client -> Proxy would be via the selected URL. Each proxy would also
have a
’primary’ URL which Foreman would always use to connect to it via.

Does this approach make sense?
Any feedback to greatly appreciated

Changes required:
Foreman: https://github.com/theforeman/foreman/pull/4346

  • Adds Proxy URL concept
  • Puppet CA & Puppet Master selection are now URLS

Katello: PR to be raised
Content Source selection need to list URLS.
Bootstrap RPM will need to create one for each URL

Let me know what you think or any questions you have!
Sean

9

> >
> > > > > Can you elaborate more on motivation? Multi-homed proxy is still
> one
> > > > > instance, why would
> > > > > Foreman need to reach to it in two different ways?
> > > >
> > > > This is for clients to reach a proxy in the different ways, NOT
> foreman.
> > > >
> > > >
> > > > > Isn't the result always the same (a change on the same instance)?
> How
> > > > > about relaxing
> > > > > the constraing when each individual proxy must have unique URL?
> Just
> > > > > checking before introducing another table (more SQL joins and query
> > > > > complexity).
> > > > >
> > > > If a Host / HostGroups can be assigned a URL/Hostname instead of a
> proxy
> > > > its useful for Multi-homing, loadbalancing & NAT'ed environments,
> maybe
> > > > others?
> > > > What does relaxing the unique URL constrain actually give us? I
> think if
> > > > you have 2 proxies with the same URL that's just effectively
> duplicating
> > > > the object?
> > > >
> > > > Or if you mean creating 2 proxies with different URLs (an external &
> and
> > > > another internal one) I think would be problematic when Foreman
> needs to
> > > > talk to the Proxy. E.g Host needs to talk to
> proxy1-external.example.com
> > > > but foreman must talk to the proxy via proxy1-internal.example.com.
> If
> > > we
> > > > have 2 proxies in foreman ( proxy1-internal & proxy1-external ) and
> you
> > > > assign a host proxy1-external foreman would try to connect to
> > > > proxy1-external.example.com to create a DHCP or DNS or something
> else
> > > with
> > > > would not work.
> > >
> > > I think a different URL for clients and Foreman itself makes a lot of
> > > sense. Now I have some trouble reading the design from the code. Could
> > > you describe how this would be used by a user? Maybe a screenshot from
> > > the new configuration page.
> > >
> >
> > Sure,
> >
> > I've added another tab to the Smart Proxies form, where you can add
> > "Secondary URLs". On the Host/Hostgroup from the user can then select a
> URL
> > for attributes like Puppet Master, Puppet CA (and Openscap and content
> > source for those plugins). I've uploaded some pictures to the PR as well
> > which should explain it much better.
> >
> > https://github.com/theforeman/foreman/pull/4346#issuecomment-288461410
> >
> > One thing Stephen mentioned on the PR was if URLs could have many proxies
> > as well as proxies have many URLs. That would go someway to better load
> > balanced proxies. Which I think would be great, though I personally don't
> > like the idea of another form & menu item just to create URLs, I think we
> > have lots of forms and menu items already…
>
> As a user it feels very complex and overwhelming. This is a perfect
> example where a use case for a complex deployment makes it harder for
> simple deployments.
>
> I'm not against it, but just some thinking out loud to see if we both
> get what the impact is.
>
> How often do you have multiple URLs? My feeling is that there are the
> following use cases:
>
> # A proxy has 1 URL
>
> This is the current situation. Speaks for itself. Question here is how
> you display a proxy with only 1 URL. Do you simplify it so it matches
> the current UI or keep it consistent with multiple URLs?
>

I would say keeping the UI consistent with multiple URLs would be better,
as it would look funny IMO if you have grouped options listing
URLs/hostname and ungrouped options listing proxies names.
maybe Roxanna or someone else has some input on this?

Maybe we could:
Just listing the Hostname?
or if the URL has a Name attribute they could select a Name? Something like
"Atlanta - Dev Network". @stbenjam suggested that on the PR.

> # A proxy has 2 URLs
>
> Here we have a separate network for Foreman and its proxies. Then the
> clients talk to their proxies on another network for other services
> (puppet, DHCP etc).
> That means we have a URL for the internal network and a URL for the
> client network.
>
> # A proxy has multiple URLs for multiple clients
>
> A proxy could live in multiple client networks and serve them with
> different names. I'm wondering how common this is and it's really an
> added benefit. If you're separating anyway, maybe we should just
> recommend deploying multiple proxies.
>

That's the currently way, but when you have lots of networks you end up
having to have lots of unnecessary Proxies on small networks that otherwise
wouldn't need it.

>
> Regarding the use case where a URL has multiple proxies: this is a proxy
> cluster. You could reason the other way around. Let me explain that.
> Currently a proxy has multiple features. Each feature is reachable under
> the same URL as a proxy. What you could consider is that you have the
> proxy feature class.
>
> ProxyFeature
> * name (puppet, tftp)
> * URL
> * foreman proxies (1 or more)
>
> Now you do need some logic to know if Foreman needs to talk to 1 of the
> proxies or all. In the case of importing puppet classes you just need 1,
> but in case of tftp all need to download. Some features might not allow
> multiple proxies.
>
> One complexity might be that not everything is a URL. tftp is just an IP
> or hostname.
>
> Does that make sense?
>

Kind of, but I don't really like it to be honest. Storing a URL N times for
every feature seems unnecessary to me? If we need the IP then we could get
that from a lookup using the hostname, which I think is how its done
currently. Though yes, it is a valid point some features wouldn't be able
to make use of all URLs, I would consider these edge cases (as most would
'JustWork'™) and we can document which features work and don't work with
multiple interfaces. DHCP for example - I think the installer only
currently supports 1 interface, but not everyone uses DHCP.

*It'd be really nice to get other people input on this as well! Even a
simple +1 or -1 with a short sentence would be appreciated :slight_smile: *
Especially guys from OpenSCAP, Katello & Remote EX plugins…

··· On Mon, Mar 27, 2017 at 3:10 PM, Ewoud Kohl van Wijngaarden < ewoud@kohlvanwijngaarden.nl> wrote: > On Wed, Mar 22, 2017 at 04:44:51PM +0000, Sean O'Keeffe wrote: > > On Wed, Mar 22, 2017 at 1:40 PM, Ewoud Kohl van Wijngaarden < > > ewoud@kohlvanwijngaarden.nl> wrote: > > > On Mon, Mar 20, 2017 at 03:50:30PM +0000, Sean O'Keeffe wrote: > > > > On Mon, Mar 20, 2017 at 2:25 PM, Lukas Zapletal > wrote:

On Wed, Mar 15, 2017 at 11:18 AM, Sean O’Keeffe < > sokeeffe@redhat.com> > > > > > wrote:

Hey!

So lzap suggested this… lets see how it goes :slight_smile:

I’m looking to solve the following issues

  • I would like to have a multi-homed proxy.
  • I would like clients to connect to a Proxy when there is a NAT
    between.

My approach has been for proxies to have multiple URLs and during
provisioning the user would then select a URL instead of the
current

method

of selecting a Proxy, This would be then mean all communication
between

the

Client -> Proxy would be via the selected URL. Each proxy would
also

have a

‘primary’ URL which Foreman would always use to connect to it
via.

Does this approach make sense?
Any feedback to greatly appreciated

Changes required:
Foreman: https://github.com/theforeman/foreman/pull/4346

  • Adds Proxy URL concept
  • Puppet CA & Puppet Master selection are now URLS

Katello: PR to be raised
Content Source selection need to list URLS.
Bootstrap RPM will need to create one for each URL

Let me know what you think or any questions you have!
Sean

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

10

I can't speak to the technical aspects here, but I don't have a problem
with the grouping you showed in the PR for the dropdown. That said, you
mentioned potentially listing by Hostname or Name attribute, I would ask -
what method would a user be most familiar with? I took a look in the demo
environment a do have to see if there is currently any grouping patterns
for URLs (when they are in a dropdown) and I didn't discover any (that were
grouped).

··· On Wednesday, March 29, 2017 at 6:57:57 AM UTC-4, Sean O'Keeffe wrote: > > > > On Mon, Mar 27, 2017 at 3:10 PM, Ewoud Kohl van Wijngaarden < > ew...@kohlvanwijngaarden.nl > wrote: > >> On Wed, Mar 22, 2017 at 04:44:51PM +0000, Sean O'Keeffe wrote: >> > On Wed, Mar 22, 2017 at 1:40 PM, Ewoud Kohl van Wijngaarden < >> > ew...@kohlvanwijngaarden.nl > wrote: >> > >> > > On Mon, Mar 20, 2017 at 03:50:30PM +0000, Sean O'Keeffe wrote: >> > > > On Mon, Mar 20, 2017 at 2:25 PM, Lukas Zapletal > > wrote: >> > > > > Can you elaborate more on motivation? Multi-homed proxy is still >> one >> > > > > instance, why would >> > > > > Foreman need to reach to it in two different ways? >> > > > >> > > > This is for clients to reach a proxy in the different ways, NOT >> foreman. >> > > > >> > > > >> > > > > Isn't the result always the same (a change on the same instance)? >> How >> > > > > about relaxing >> > > > > the constraing when each individual proxy must have unique URL? >> Just >> > > > > checking before introducing another table (more SQL joins and >> query >> > > > > complexity). >> > > > > >> > > > If a Host / HostGroups can be assigned a URL/Hostname instead of a >> proxy >> > > > its useful for Multi-homing, loadbalancing & NAT'ed environments, >> maybe >> > > > others? >> > > > What does relaxing the unique URL constrain actually give us? I >> think if >> > > > you have 2 proxies with the same URL that's just effectively >> duplicating >> > > > the object? >> > > > >> > > > Or if you mean creating 2 proxies with different URLs (an external >> & and >> > > > another internal one) I think would be problematic when Foreman >> needs to >> > > > talk to the Proxy. E.g Host needs to talk to >> proxy1-external.example.com >> > > > but foreman must talk to the proxy via proxy1-internal.example.com. >> If >> > > we >> > > > have 2 proxies in foreman ( proxy1-internal & proxy1-external ) and >> you >> > > > assign a host proxy1-external foreman would try to connect to >> > > > proxy1-external.example.com to create a DHCP or DNS or something >> else >> > > with >> > > > would not work. >> > > >> > > I think a different URL for clients and Foreman itself makes a lot of >> > > sense. Now I have some trouble reading the design from the code. Could >> > > you describe how this would be used by a user? Maybe a screenshot from >> > > the new configuration page. >> > > >> > >> > Sure, >> > >> > I've added another tab to the Smart Proxies form, where you can add >> > "Secondary URLs". On the Host/Hostgroup from the user can then select a >> URL >> > for attributes like Puppet Master, Puppet CA (and Openscap and content >> > source for those plugins). I've uploaded some pictures to the PR as well >> > which should explain it much better. >> > >> > https://github.com/theforeman/foreman/pull/4346#issuecomment-288461410 >> > >> > One thing Stephen mentioned on the PR was if URLs could have many >> proxies >> > as well as proxies have many URLs. That would go someway to better load >> > balanced proxies. Which I think would be great, though I personally >> don't >> > like the idea of another form & menu item just to create URLs, I think >> we >> > have lots of forms and menu items already.. >> >> As a user it feels very complex and overwhelming. This is a perfect >> example where a use case for a complex deployment makes it harder for >> simple deployments. >> >> I'm not against it, but just some thinking out loud to see if we both >> get what the impact is. >> >> How often do you have multiple URLs? My feeling is that there are the >> following use cases: >> >> # A proxy has 1 URL >> >> This is the current situation. Speaks for itself. Question here is how >> you display a proxy with only 1 URL. Do you simplify it so it matches >> the current UI or keep it consistent with multiple URLs? >> > > I would say keeping the UI consistent with multiple URLs would be better, > as it would look funny IMO if you have grouped options listing > URLs/hostname and ungrouped options listing proxies names. > maybe Roxanna or someone else has some input on this? > > Maybe we could: > Just listing the Hostname? > or if the URL has a Name attribute they could select a Name? Something > like "Atlanta - Dev Network". @stbenjam suggested that on the PR. > > >> # A proxy has 2 URLs >> >> Here we have a separate network for Foreman and its proxies. Then the >> clients talk to their proxies on another network for other services >> (puppet, DHCP etc). >> That means we have a URL for the internal network and a URL for the >> client network. >> >> # A proxy has multiple URLs for multiple clients >> >> A proxy could live in multiple client networks and serve them with >> different names. I'm wondering how common this is and it's really an >> added benefit. If you're separating anyway, maybe we should just >> recommend deploying multiple proxies. >> > > That's the currently way, but when you have lots of networks you end up > having to have lots of unnecessary Proxies on small networks that otherwise > wouldn't need it. > > >> >> Regarding the use case where a URL has multiple proxies: this is a proxy >> cluster. You could reason the other way around. Let me explain that. >> Currently a proxy has multiple features. Each feature is reachable under >> the same URL as a proxy. What you could consider is that you have the >> proxy feature class. >> >> ProxyFeature >> * name (puppet, tftp) >> * URL >> * foreman proxies (1 or more) >> >> Now you do need some logic to know if Foreman needs to talk to 1 of the >> proxies or all. In the case of importing puppet classes you just need 1, >> but in case of tftp all need to download. Some features might not allow >> multiple proxies. >> >> One complexity might be that not everything is a URL. tftp is just an IP >> or hostname. >> >> Does that make sense? >> > > Kind of, but I don't really like it to be honest. Storing a URL N times > for every feature seems unnecessary to me? If we need the IP then we could > get that from a lookup using the hostname, which I think is how its done > currently. Though yes, it is a valid point some features wouldn't be able > to make use of all URLs, I would consider these edge cases (as most would > 'JustWork'™) and we can document which features work and don't work with > multiple interfaces. DHCP for example - I think the installer only > currently supports 1 interface, but not everyone uses DHCP. > > > *It'd be really nice to get other people input on this as well! Even a > simple +1 or -1 with a short sentence would be appreciated :-) * > Especially guys from OpenSCAP, Katello & Remote EX plugins... > > >> > > >> > > > > On Wed, Mar 15, 2017 at 11:18 AM, Sean O'Keeffe < >> soke...@redhat.com > >> > > > > wrote: >> > > > > > Hey! >> > > > > > >> > > > > > So lzap suggested this.. lets see how it goes :-) >> > > > > > >> > > > > > I'm looking to solve the following issues >> > > > > > - I would like to have a multi-homed proxy. >> > > > > > - I would like clients to connect to a Proxy when there is a NAT >> > > between. >> > > > > > >> > > > > > My approach has been for proxies to have multiple URLs and >> during >> > > > > > provisioning the user would then select a URL instead of the >> current >> > > > > method >> > > > > > of selecting a Proxy, This would be then mean all communication >> > > between >> > > > > the >> > > > > > Client -> Proxy would be via the selected URL. Each proxy would >> also >> > > > > have a >> > > > > > 'primary' URL which Foreman would always use to connect to it >> via. >> > > > > > >> > > > > > Does this approach make sense? >> > > > > > Any feedback to greatly appreciated >> > > > > > >> > > > > > Changes required: >> > > > > > Foreman: https://github.com/theforeman/foreman/pull/4346 >> > > > > > - Adds Proxy URL concept >> > > > > > - Puppet CA & Puppet Master selection are now URLS >> > > > > > >> > > > > > Katello: PR to be raised >> > > > > > Content Source selection need to list URLS. >> > > > > > Bootstrap RPM will need to create one for each URL >> > > > > > >> > > > > > Let me know what you think or any questions you have! >> > > > > > Sean >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev...@googlegroups.com . >> For more options, visit https://groups.google.com/d/optout. >> > >
11

@sshtein I think requiring defined subnets would be a bad idea, for example
if you're using foreman_discovery you don't need to define subnets at all.
Also storing a hostname in the smart_proxies table is kind of weird IMO,
having an attribute to indicate that its actually a hostname of this proxy
doesn't feel right. Also what about if its a hostname for multiple proxies?
would we have a comma separated list of ids? that feels very wrong, this is
what Active Record Associations are for

I think I'll redo this as the following unless anyone has any objections… I
know we don't really want to include another join but i don't see any other
way to cater for LoadBalanced & Multi-homed Smart Proxies.

  • A new SmartProxyHostname model (with name & hostname attrs) with a
    has_and_belongs_to_many relationship to SmartProxy and vise versa.
  • A new Smart Proxy Hostnames entry under the "Infrastructure" menu item.
  • On Smart Proxy creation a Hostname would also be created from the URL
    with the same Name as the Smart Proxy (so complexity isn't increased for
    simple deployments)
  • Hosts & Hostgroups objects would select a Hostname by Name. (this would
    be transparent as by default the Hostname would have the same name as the
    Smart Proxy)

Comments & suggestions are very welcome

··· On Tue, Apr 4, 2017 at 8:20 PM, Roxanne Hoover wrote:

I can’t speak to the technical aspects here, but I don’t have a problem
with the grouping you showed in the PR for the dropdown. That said, you
mentioned potentially listing by Hostname or Name attribute, I would ask -
what method would a user be most familiar with? I took a look in the demo
environment a do have to see if there is currently any grouping patterns
for URLs (when they are in a dropdown) and I didn’t discover any (that were
grouped).

On Wednesday, March 29, 2017 at 6:57:57 AM UTC-4, Sean O’Keeffe wrote:

On Mon, Mar 27, 2017 at 3:10 PM, Ewoud Kohl van Wijngaarden < >> ew...@kohlvanwijngaarden.nl> wrote:

On Wed, Mar 22, 2017 at 04:44:51PM +0000, Sean O’Keeffe wrote:

On Wed, Mar 22, 2017 at 1:40 PM, Ewoud Kohl van Wijngaarden < >>> > ew...@kohlvanwijngaarden.nl> wrote:

On Mon, Mar 20, 2017 at 03:50:30PM +0000, Sean O’Keeffe wrote:

On Mon, Mar 20, 2017 at 2:25 PM, Lukas Zapletal lz...@redhat.com >>> wrote:

Can you elaborate more on motivation? Multi-homed proxy is still
one

instance, why would
Foreman need to reach to it in two different ways?

This is for clients to reach a proxy in the different ways, NOT
foreman.

Isn’t the result always the same (a change on the same
instance)? How

about relaxing
the constraing when each individual proxy must have unique URL?
Just

checking before introducing another table (more SQL joins and
query

complexity).

If a Host / HostGroups can be assigned a URL/Hostname instead of a
proxy

its useful for Multi-homing, loadbalancing & NAT’ed environments,
maybe

others?
What does relaxing the unique URL constrain actually give us? I
think if

you have 2 proxies with the same URL that’s just effectively
duplicating

the object?

Or if you mean creating 2 proxies with different URLs (an external
& and

another internal one) I think would be problematic when Foreman
needs to

talk to the Proxy. E.g Host needs to talk to
proxy1-external.example.com

but foreman must talk to the proxy via proxy1-internal.example.com.
If

we

have 2 proxies in foreman ( proxy1-internal & proxy1-external )
and you

assign a host proxy1-external foreman would try to connect to
proxy1-external.example.com to create a DHCP or DNS or something
else

with

would not work.

I think a different URL for clients and Foreman itself makes a lot of
sense. Now I have some trouble reading the design from the code.
Could

you describe how this would be used by a user? Maybe a screenshot
from

the new configuration page.

Sure,

I’ve added another tab to the Smart Proxies form, where you can add
"Secondary URLs". On the Host/Hostgroup from the user can then select
a URL
for attributes like Puppet Master, Puppet CA (and Openscap and content
source for those plugins). I’ve uploaded some pictures to the PR as
well
which should explain it much better.

https://github.com/theforeman/foreman/pull/4346#issuecomment-288461410

One thing Stephen mentioned on the PR was if URLs could have many
proxies
as well as proxies have many URLs. That would go someway to better load
balanced proxies. Which I think would be great, though I personally
don’t
like the idea of another form & menu item just to create URLs, I think
we
have lots of forms and menu items already…

As a user it feels very complex and overwhelming. This is a perfect
example where a use case for a complex deployment makes it harder for
simple deployments.

I’m not against it, but just some thinking out loud to see if we both
get what the impact is.

How often do you have multiple URLs? My feeling is that there are the
following use cases:

A proxy has 1 URL

This is the current situation. Speaks for itself. Question here is how
you display a proxy with only 1 URL. Do you simplify it so it matches
the current UI or keep it consistent with multiple URLs?

I would say keeping the UI consistent with multiple URLs would be better,
as it would look funny IMO if you have grouped options listing
URLs/hostname and ungrouped options listing proxies names.
maybe Roxanna or someone else has some input on this?

Maybe we could:
Just listing the Hostname?
or if the URL has a Name attribute they could select a Name? Something
like “Atlanta - Dev Network”. @stbenjam suggested that on the PR.

A proxy has 2 URLs

Here we have a separate network for Foreman and its proxies. Then the
clients talk to their proxies on another network for other services
(puppet, DHCP etc).
That means we have a URL for the internal network and a URL for the
client network.

A proxy has multiple URLs for multiple clients

A proxy could live in multiple client networks and serve them with
different names. I’m wondering how common this is and it’s really an
added benefit. If you’re separating anyway, maybe we should just
recommend deploying multiple proxies.

That’s the currently way, but when you have lots of networks you end up
having to have lots of unnecessary Proxies on small networks that otherwise
wouldn’t need it.

Regarding the use case where a URL has multiple proxies: this is a proxy
cluster. You could reason the other way around. Let me explain that.
Currently a proxy has multiple features. Each feature is reachable under
the same URL as a proxy. What you could consider is that you have the
proxy feature class.

ProxyFeature

  • name (puppet, tftp)
  • URL
  • foreman proxies (1 or more)

Now you do need some logic to know if Foreman needs to talk to 1 of the
proxies or all. In the case of importing puppet classes you just need 1,
but in case of tftp all need to download. Some features might not allow
multiple proxies.

One complexity might be that not everything is a URL. tftp is just an IP
or hostname.

Does that make sense?

Kind of, but I don’t really like it to be honest. Storing a URL N times
for every feature seems unnecessary to me? If we need the IP then we could
get that from a lookup using the hostname, which I think is how its done
currently. Though yes, it is a valid point some features wouldn’t be able
to make use of all URLs, I would consider these edge cases (as most would
’JustWork’™) and we can document which features work and don’t work with
multiple interfaces. DHCP for example - I think the installer only
currently supports 1 interface, but not everyone uses DHCP.

*It’d be really nice to get other people input on this as well! Even a
simple +1 or -1 with a short sentence would be appreciated :slight_smile: *
Especially guys from OpenSCAP, Katello & Remote EX plugins…

On Wed, Mar 15, 2017 at 11:18 AM, Sean O’Keeffe < >>> soke...@redhat.com> >>> >>> > > > > wrote:

Hey!

So lzap suggested this… lets see how it goes :slight_smile:

I’m looking to solve the following issues

  • I would like to have a multi-homed proxy.
  • I would like clients to connect to a Proxy when there is a
    NAT

between.

My approach has been for proxies to have multiple URLs and
during

provisioning the user would then select a URL instead of the
current

method

of selecting a Proxy, This would be then mean all communication
between

the

Client -> Proxy would be via the selected URL. Each proxy
would also

have a

‘primary’ URL which Foreman would always use to connect to it
via.

Does this approach make sense?
Any feedback to greatly appreciated

Changes required:
Foreman: https://github.com/theforeman/foreman/pull/4346

  • Adds Proxy URL concept
  • Puppet CA & Puppet Master selection are now URLS

Katello: PR to be raised
Content Source selection need to list URLS.
Bootstrap RPM will need to create one for each URL

Let me know what you think or any questions you have!
Sean

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

12

Given that Hosts can already be multihomed, and have a name per nic,
does itmake sense to store this in the Interfaces table, instead of
creating a new table? If the code is sufficiently different, the
Interfaces model is STId too, so you could create an Interface::Proxy
model if need be.

I think that would get you want you need, no? UI side sounds sane
enough, as do the defaults, it'd just mean re-using the code we already
have for things with multiple names.

Greg

··· On 04/05/17 17:22, Sean O'Keeffe wrote:

I think I’ll redo this as the following unless anyone has any
objections… I know we don’t really want to include another join but i
don’t see any other way to cater for LoadBalanced & Multi-homed Smart
Proxies.

  • A new SmartProxyHostname model (with name & hostname attrs) with a
    has_and_belongs_to_many relationship to SmartProxy and vise versa.
  • A new Smart Proxy Hostnames entry under the “Infrastructure” menu item.
  • On Smart Proxy creation a Hostname would also be created from the URL
    with the same Name as the Smart Proxy (so complexity isn’t increased for
    simple deployments)
  • Hosts & Hostgroups objects would select a Hostname by Name. (this
    would be transparent as by default the Hostname would have the same name
    as the Smart Proxy)