Hej,
at the moment there is a PR[1] open on foreman-docker to set a HTTP proxy
for requests to registries.
The PR allows to set a HTTP proxy on the HTTP client, in this case deep
down Excon, only for registry requests.
A HTTP proxy won't be set on requests if a HTTP_PROXY environment
variable is available, since it is an unlikely setup to have registry
request routed over a different proxy than other requests. However setting
it via the environment variable will allow requests to succeed to resources
available by the HTTP proxy, but will fail for those inside and possible
blocked.
The HTTP_PROXY environment variable seems to be a standard, and therefore
Excon is built to use it when available.
Excon is used by docker-api as well as fog, it might be used by other
components and there might be other parts that use another HTTP client like
RestClient, which also respects the variable.
This means at the moment with that environment variable set some requests
would already rely on it.
In any case this should be in mentioned in the manual to be aware of, also
because some operating systems set this globally.
The question is should we make an afford to ensure deployment behind a HTTP
proxy on a system with HTTP blocked works without issues and provide a way
to configure it properly?
I've tested Foreman with HTTP blocked and HTTP_PROXY set, but in a very
basic setup, with the only external requests being to Docker registries
outside and squid configured to just pass requests through regardless there
to.
It didn't show any apparent issue, but there are for sure issues with a
more robust configured HTTP proxy.
This raises another question: How common is a setup where external
resources requiring HTTP are used with Foreman behind a HTTP proxy?
Comments?
All the best,
Sebastian
[1] https://github.com/theforeman/foreman-docker/pull/189
Hi,
> How common is a setup where external resources requiring HTTP are used
> with Foreman behind a HTTP proxy?
>
I believe, this is very common in enterprise environments. Usually any
internet access is blocked for security reasons and only connections via
a proxy server are allowed. The proxy ususally does a MITM attack to be
able to investigate encrypted traffic. While this does make sense in
some cases, don't get me started why it does not make any sense in others.
> Comments?
I think especially access to all the docker registries out there is
something a corporate it-security team would want to go through a proxy
server. Setting a proxy server on a server via environment variables
(http_proxy) is quite easy with systemd unit files. However that may
lead to problems when the client doesn't respect the 'no_proxy'
environment variable and suddenly all requests to a smart proxy are
routed via the http proxy. This is problematic when smart-proxy is on
the same network and not reachable via the proxy server.
I personally prefer only to have an explicit option in a settings file.
···
Am 20.04.17 um 13:06 schrieb Sebastian Gräßl:
There was some feedback regarding this on the PR[1] mentioned in the
beginning.
There is already a RFC[2] regarding this and a plugin[3] to implement the
solution proposed in the RFC.
The solution proposed by jlsherrill allows to add multiple HTTP-proxies in
Foreman and use these in plugins and allow to configure what HTTP-proxy
should be used for what requests.
So far the plugin only adds the ability to add HTTP proxies and misses a
essential part, which is applying the HTTP proxies to requests.
While looking at how other applications handle this and also considering
typical HTTP proxy configurations, it feels that such a solution would make
it rather complex in practice to apply. Configuring rules for requests or
just ensuring the proper request is using the right HTTP proxy is better
configurable in the HTTP proxy itself.
I believe a very bold simple solution would be the better, which in my
opinion would be to ensure all parts respect a HTTP proxy configuration and
have good documentation around it to advice on how to configure the HTTP
proxy correctly. Taken other applications in the same area the HTTP_PROXY
environment variable seems to be the common standard to use.
Please, I would love to hear more feedback on this and what are common HTTP
proxy setups.
[1] https://github.com/theforeman/foreman-docker/pull/189
[2] https://github.com/theforeman/rfcs/pull/18
[3] https://github.com/jlsherrill/foreman_http_proxies
···
On Thursday, April 20, 2017 at 1:07:33 PM UTC+2, Sebastian Gräßl wrote:
>
> Hej,
>
> at the moment there is a PR[1] open on foreman-docker to set a HTTP proxy
> for requests to registries.
> The PR allows to set a HTTP proxy on the HTTP client, in this case deep
> down Excon, only for registry requests.
>
> A HTTP proxy won't be set on requests if a `HTTP_PROXY` environment
> variable is available, since it is an unlikely setup to have registry
> request routed over a different proxy than other requests. However setting
> it via the environment variable will allow requests to succeed to resources
> available by the HTTP proxy, but will fail for those inside and possible
> blocked.
>
> The `HTTP_PROXY` environment variable seems to be a standard, and
> therefore Excon is built to use it when available.
> Excon is used by docker-api as well as fog, it might be used by other
> components and there might be other parts that use another HTTP client like
> RestClient, which also respects the variable.
>
> This means at the moment with that environment variable set some requests
> would already rely on it.
> In any case this should be in mentioned in the manual to be aware of, also
> because some operating systems set this globally.
>
> The question is should we make an afford to ensure deployment behind a
> HTTP proxy on a system with HTTP blocked works without issues and provide a
> way to configure it properly?
>
> I've tested Foreman with HTTP blocked and `HTTP_PROXY` set, but in a very
> basic setup, with the only external requests being to Docker registries
> outside and squid configured to just pass requests through regardless there
> to.
>
> It didn't show any apparent issue, but there are for sure issues with a
> more robust configured HTTP proxy.
> This raises another question: How common is a setup where external
> resources requiring HTTP are used with Foreman behind a HTTP proxy?
>
> Comments?
>
> All the best,
> Sebastian
>
> [1] https://github.com/theforeman/foreman-docker/pull/189
>
>
As a dev I am curious how this is handled in our code. Does every usage of
RestClient need to know about the existence of a proxy? Are there multiple
proxies for different aspects (eg. https://cdn.redhat.com vs.
https://myinternalserver.example.com)?
···
On Thu, Apr 20, 2017 at 8:43 AM, Timo Goebel wrote:
Hi,
Am 20.04.17 um 13:06 schrieb Sebastian Gräßl:
How common is a setup where external resources requiring HTTP are used
with Foreman behind a HTTP proxy?
I believe, this is very common in enterprise environments. Usually any
internet access is blocked for security reasons and only connections via a
proxy server are allowed. The proxy ususally does a MITM attack to be able
to investigate encrypted traffic. While this does make sense in some cases,
don’t get me started why it does not make any sense in others.
Comments?
I think especially access to all the docker registries out there is
something a corporate it-security team would want to go through a proxy
server. Setting a proxy server on a server via environment variables
(http_proxy) is quite easy with systemd unit files. However that may lead
to problems when the client doesn’t respect the ‘no_proxy’ environment
variable and suddenly all requests to a smart proxy are routed via the http
proxy. This is problematic when smart-proxy is on the same network and not
reachable via the proxy server.
I personally prefer only to have an explicit option in a settings file.
–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
After reading the RFC I think that more robust and adaptable solution would
be better. A single env var is not going to cover the needs of all the
scenarios. A simple example may be accessing both registry.access.redhat.com
(through proxy) and myopenshift:5000 (no proxy).
As @jlsherrill noted on the PR, the temporary solution for the
foreman-docker plugin is alright for the moment.
···
On Wed, May 17, 2017 at 3:08 AM, Sebastian Gräßl wrote:
There was some feedback regarding this on the PR[1] mentioned in the
beginning.
There is already a RFC[2] regarding this and a plugin[3] to implement the
solution proposed in the RFC.
The solution proposed by jlsherrill allows to add multiple HTTP-proxies in
Foreman and use these in plugins and allow to configure what HTTP-proxy
should be used for what requests.
So far the plugin only adds the ability to add HTTP proxies and misses a
essential part, which is applying the HTTP proxies to requests.
While looking at how other applications handle this and also considering
typical HTTP proxy configurations, it feels that such a solution would make
it rather complex in practice to apply. Configuring rules for requests or
just ensuring the proper request is using the right HTTP proxy is better
configurable in the HTTP proxy itself.
I believe a very bold simple solution would be the better, which in my
opinion would be to ensure all parts respect a HTTP proxy configuration and
have good documentation around it to advice on how to configure the HTTP
proxy correctly. Taken other applications in the same area the HTTP_PROXY
environment variable seems to be the common standard to use.
Please, I would love to hear more feedback on this and what are common
HTTP proxy setups.
[1] https://github.com/theforeman/foreman-docker/pull/189
[2] https://github.com/theforeman/rfcs/pull/18
[3] https://github.com/jlsherrill/foreman_http_proxies
On Thursday, April 20, 2017 at 1:07:33 PM UTC+2, Sebastian Gräßl wrote:
Hej,
at the moment there is a PR[1] open on foreman-docker to set a HTTP proxy
for requests to registries.
The PR allows to set a HTTP proxy on the HTTP client, in this case deep
down Excon, only for registry requests.
A HTTP proxy won’t be set on requests if a HTTP_PROXY environment
variable is available, since it is an unlikely setup to have registry
request routed over a different proxy than other requests. However setting
it via the environment variable will allow requests to succeed to resources
available by the HTTP proxy, but will fail for those inside and possible
blocked.
The HTTP_PROXY environment variable seems to be a standard, and
therefore Excon is built to use it when available.
Excon is used by docker-api as well as fog, it might be used by other
components and there might be other parts that use another HTTP client like
RestClient, which also respects the variable.
This means at the moment with that environment variable set some requests
would already rely on it.
In any case this should be in mentioned in the manual to be aware of,
also because some operating systems set this globally.
The question is should we make an afford to ensure deployment behind a
HTTP proxy on a system with HTTP blocked works without issues and provide a
way to configure it properly?
I’ve tested Foreman with HTTP blocked and HTTP_PROXY set, but in a very
basic setup, with the only external requests being to Docker registries
outside and squid configured to just pass requests through regardless there
to.
It didn’t show any apparent issue, but there are for sure issues with a
more robust configured HTTP proxy.
This raises another question: How common is a setup where external
resources requiring HTTP are used with Foreman behind a HTTP proxy?
Comments?
All the best,
Sebastian
[1] https://github.com/theforeman/foreman-docker/pull/189
–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
> After reading the RFC I think that more robust and adaptable solution
> would be better. A single env var is not going to cover the needs of
> all the scenarios. A simple example may be accessing both
> registry.access.redhat.com <http://registry.access.redhat.com>
> (through proxy) and myopenshift:5000 (no proxy).
>
> As @jlsherrill noted on the PR, the temporary solution for the
> foreman-docker plugin is alright for the moment.
I'd like to echo what tom said, we've had many users that want to access
content externally through a proxy and internally (where the proxy is
not controlled by them and does not properly proxy internal requests).
Its happened enough for me to say that a simple solution is not good
enough long term.
···
On 05/17/2017 07:57 AM, Tom McKay wrote:
On Wed, May 17, 2017 at 3:08 AM, Sebastian Gräßl > <sebastian@validcode.me mailto:sebastian@validcode.me> wrote:
There was some feedback regarding this on the PR[1] mentioned in
the beginning.
There is already a RFC[2] regarding this and a plugin[3] to
implement the solution proposed in the RFC.
The solution proposed by jlsherrill allows to add multiple
HTTP-proxies in Foreman and use these in plugins and allow to
configure what HTTP-proxy should be used for what requests.
So far the plugin only adds the ability to add HTTP proxies and
misses a essential part, which is applying the HTTP proxies to
requests.
While looking at how other applications handle this and also
considering typical HTTP proxy configurations, it feels that such
a solution would make it rather complex in practice to apply.
Configuring rules for requests or just ensuring the proper request
is using the right HTTP proxy is better configurable in the HTTP
proxy itself.
I believe a very bold simple solution would be the better, which
in my opinion would be to ensure all parts respect a HTTP proxy
configuration and have good documentation around it to advice on
how to configure the HTTP proxy correctly. Taken other
applications in the same area the HTTP_PROXY environment variable
seems to be the common standard to use.
Please, I would love to hear more feedback on this and what are
common HTTP proxy setups.
[1] https://github.com/theforeman/foreman-docker/pull/189
<https://github.com/theforeman/foreman-docker/pull/189>
[2] https://github.com/theforeman/rfcs/pull/18
<https://github.com/theforeman/rfcs/pull/18>
[3] https://github.com/jlsherrill/foreman_http_proxies
<https://github.com/jlsherrill/foreman_http_proxies>
On Thursday, April 20, 2017 at 1:07:33 PM UTC+2, Sebastian Gräßl > wrote:
Hej,
at the moment there is a PR[1] open on foreman-docker to set a
HTTP proxy for requests to registries.
The PR allows to set a HTTP proxy on the HTTP client, in this
case deep down Excon, only for registry requests.
A HTTP proxy won't be set on requests if a `HTTP_PROXY`
environment variable is available, since it is an unlikely
setup to have registry request routed over a different proxy
than other requests. However setting it via the environment
variable will allow requests to succeed to resources available
by the HTTP proxy, but will fail for those inside and possible
blocked.
The `HTTP_PROXY` environment variable seems to be a standard,
and therefore Excon is built to use it when available.
Excon is used by docker-api as well as fog, it might be used
by other components and there might be other parts that use
another HTTP client like RestClient, which also respects the
variable.
This means at the moment with that environment variable set
some requests would already rely on it.
In any case this should be in mentioned in the manual to be
aware of, also because some operating systems set this globally.
The question is should we make an afford to ensure deployment
behind a HTTP proxy on a system with HTTP blocked works
without issues and provide a way to configure it properly?
I've tested Foreman with HTTP blocked and `HTTP_PROXY` set,
but in a very basic setup, with the only external requests
being to Docker registries outside and squid configured to
just pass requests through regardless there to.
It didn't show any apparent issue, but there are for sure
issues with a more robust configured HTTP proxy.
This raises another question: How common is a setup where
external resources requiring HTTP are used with Foreman behind
a HTTP proxy?
Comments?
All the best,
Sebastian
[1] https://github.com/theforeman/foreman-docker/pull/189
<https://github.com/theforeman/foreman-docker/pull/189>
--
You received this message because you are subscribed to the Google
Groups "foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to foreman-dev+unsubscribe@googlegroups.com
<mailto:foreman-dev+unsubscribe@googlegroups.com>.
For more options, visit https://groups.google.com/d/optout
<https://groups.google.com/d/optout>.
–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com
mailto:foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Sorry for being late to the party, sending my 2c:
I agree with having more complicated solution where users could have separate
proxies per service is good long-term goal. AFAIK we don't the solution atm.
Therefore I think introducing support for single, global proxy sounds as
improvement already to what we have now (nothing). What's good on this,
migrating to specific proxies should be easy, the RFC explicitly[1] mentions
it. The global proxy can have granular rules of what communication should be
passed through untouched and what should be sent through maybe other proxies.
Another advantage I see is that the global proxy offloads the configuration
from Foreman/Katello, which does not really belongs into our domain.
Later, when the RFC is implemented via foreman_http_proxies plugin, I'm happy
to stop using global proxy and improve plugins to use foreman_http_proxies if
it makes sense. It will take some time before everyone adopts it. But
meanwhile we'd still have the option to let user configure their master proxy
according to their needs.
[1] https://github.com/theforeman/rfcs/pull/18/
files#diff-12584a6580dac145ae55c2b5d67088dfR45
···
--
Marek
On středa 17. května 2017 14:22:28 CEST Justin Sherrill wrote:
On 05/17/2017 07:57 AM, Tom McKay wrote:
After reading the RFC I think that more robust and adaptable solution
would be better. A single env var is not going to cover the needs of
all the scenarios. A simple example may be accessing both
registry.access.redhat.com http://registry.access.redhat.com
(through proxy) and myopenshift:5000 (no proxy).
As @jlsherrill noted on the PR, the temporary solution for the
foreman-docker plugin is alright for the moment.
I’d like to echo what tom said, we’ve had many users that want to access
content externally through a proxy and internally (where the proxy is
not controlled by them and does not properly proxy internal requests).
Its happened enough for me to say that a simple solution is not good
enough long term.
On Wed, May 17, 2017 at 3:08 AM, Sebastian Gräßl > > > > <sebastian@validcode.me mailto:sebastian@validcode.me> wrote:
There was some feedback regarding this on the PR[1] mentioned in
the beginning.
There is already a RFC[2] regarding this and a plugin[3] to
implement the solution proposed in the RFC.
The solution proposed by jlsherrill allows to add multiple
HTTP-proxies in Foreman and use these in plugins and allow to
configure what HTTP-proxy should be used for what requests.
So far the plugin only adds the ability to add HTTP proxies and
misses a essential part, which is applying the HTTP proxies to
requests.
While looking at how other applications handle this and also
considering typical HTTP proxy configurations, it feels that such
a solution would make it rather complex in practice to apply.
Configuring rules for requests or just ensuring the proper request
is using the right HTTP proxy is better configurable in the HTTP
proxy itself.
I believe a very bold simple solution would be the better, which
in my opinion would be to ensure all parts respect a HTTP proxy
configuration and have good documentation around it to advice on
how to configure the HTTP proxy correctly. Taken other
applications in the same area the HTTP_PROXY environment variable
seems to be the common standard to use.
Please, I would love to hear more feedback on this and what are
common HTTP proxy setups.
[1] https://github.com/theforeman/foreman-docker/pull/189
<https://github.com/theforeman/foreman-docker/pull/189>
[2] https://github.com/theforeman/rfcs/pull/18
<https://github.com/theforeman/rfcs/pull/18>
[3] https://github.com/jlsherrill/foreman_http_proxies
<https://github.com/jlsherrill/foreman_http_proxies>
On Thursday, April 20, 2017 at 1:07:33 PM UTC+2, Sebastian Gräßl > > > > wrote:
Hej,
at the moment there is a PR[1] open on foreman-docker to set a
HTTP proxy for requests to registries.
The PR allows to set a HTTP proxy on the HTTP client, in this
case deep down Excon, only for registry requests.
A HTTP proxy won't be set on requests if a `HTTP_PROXY`
environment variable is available, since it is an unlikely
setup to have registry request routed over a different proxy
than other requests. However setting it via the environment
variable will allow requests to succeed to resources available
by the HTTP proxy, but will fail for those inside and possible
blocked.
The `HTTP_PROXY` environment variable seems to be a standard,
and therefore Excon is built to use it when available.
Excon is used by docker-api as well as fog, it might be used
by other components and there might be other parts that use
another HTTP client like RestClient, which also respects the
variable.
This means at the moment with that environment variable set
some requests would already rely on it.
In any case this should be in mentioned in the manual to be
aware of, also because some operating systems set this globally.
The question is should we make an afford to ensure deployment
behind a HTTP proxy on a system with HTTP blocked works
without issues and provide a way to configure it properly?
I've tested Foreman with HTTP blocked and `HTTP_PROXY` set,
but in a very basic setup, with the only external requests
being to Docker registries outside and squid configured to
just pass requests through regardless there to.
It didn't show any apparent issue, but there are for sure
issues with a more robust configured HTTP proxy.
This raises another question: How common is a setup where
external resources requiring HTTP are used with Foreman behind
a HTTP proxy?
Comments?
All the best,
Sebastian
[1] https://github.com/theforeman/foreman-docker/pull/189
<https://github.com/theforeman/foreman-docker/pull/189>
sorry for typo, it should have been:
AFAIK we don't have the solution implemented atm.
···
--
Marek
On čtvrtek 25. května 2017 11:49:24 CEST Marek Hulán wrote:
Sorry for being late to the party, sending my 2c:
I agree with having more complicated solution where users could have
separate proxies per service is good long-term goal. AFAIK we don’t the
solution atm. Therefore I think introducing support for single, global
proxy sounds as improvement already to what we have now (nothing). What’s
good on this, migrating to specific proxies should be easy, the RFC
explicitly[1] mentions it. The global proxy can have granular rules of what
communication should be passed through untouched and what should be sent
through maybe other proxies. Another advantage I see is that the global
proxy offloads the configuration from Foreman/Katello, which does not
really belongs into our domain.
Later, when the RFC is implemented via foreman_http_proxies plugin, I’m
happy to stop using global proxy and improve plugins to use
foreman_http_proxies if it makes sense. It will take some time before
everyone adopts it. But meanwhile we’d still have the option to let user
configure their master proxy according to their needs.
[1] https://github.com/theforeman/rfcs/pull/18/
files#diff-12584a6580dac145ae55c2b5d67088dfR45
–
Marek
On středa 17. května 2017 14:22:28 CEST Justin Sherrill wrote:
On 05/17/2017 07:57 AM, Tom McKay wrote:
After reading the RFC I think that more robust and adaptable solution
would be better. A single env var is not going to cover the needs of
all the scenarios. A simple example may be accessing both
registry.access.redhat.com http://registry.access.redhat.com
(through proxy) and myopenshift:5000 (no proxy).
As @jlsherrill noted on the PR, the temporary solution for the
foreman-docker plugin is alright for the moment.
I’d like to echo what tom said, we’ve had many users that want to access
content externally through a proxy and internally (where the proxy is
not controlled by them and does not properly proxy internal requests).
Its happened enough for me to say that a simple solution is not good
enough long term.
On Wed, May 17, 2017 at 3:08 AM, Sebastian Gräßl > > > > > > <sebastian@validcode.me mailto:sebastian@validcode.me> wrote:
There was some feedback regarding this on the PR[1] mentioned in
the beginning.
There is already a RFC[2] regarding this and a plugin[3] to
implement the solution proposed in the RFC.
The solution proposed by jlsherrill allows to add multiple
HTTP-proxies in Foreman and use these in plugins and allow to
configure what HTTP-proxy should be used for what requests.
So far the plugin only adds the ability to add HTTP proxies and
misses a essential part, which is applying the HTTP proxies to
requests.
While looking at how other applications handle this and also
considering typical HTTP proxy configurations, it feels that such
a solution would make it rather complex in practice to apply.
Configuring rules for requests or just ensuring the proper request
is using the right HTTP proxy is better configurable in the HTTP
proxy itself.
I believe a very bold simple solution would be the better, which
in my opinion would be to ensure all parts respect a HTTP proxy
configuration and have good documentation around it to advice on
how to configure the HTTP proxy correctly. Taken other
applications in the same area the HTTP_PROXY environment variable
seems to be the common standard to use.
Please, I would love to hear more feedback on this and what are
common HTTP proxy setups.
[1] https://github.com/theforeman/foreman-docker/pull/189
<https://github.com/theforeman/foreman-docker/pull/189>
[2] https://github.com/theforeman/rfcs/pull/18
<https://github.com/theforeman/rfcs/pull/18>
[3] https://github.com/jlsherrill/foreman_http_proxies
<https://github.com/jlsherrill/foreman_http_proxies>
On Thursday, April 20, 2017 at 1:07:33 PM UTC+2, Sebastian Gräßl > > > > > > wrote:
Hej,
at the moment there is a PR[1] open on foreman-docker to set a
HTTP proxy for requests to registries.
The PR allows to set a HTTP proxy on the HTTP client, in this
case deep down Excon, only for registry requests.
A HTTP proxy won't be set on requests if a `HTTP_PROXY`
environment variable is available, since it is an unlikely
setup to have registry request routed over a different proxy
than other requests. However setting it via the environment
variable will allow requests to succeed to resources available
by the HTTP proxy, but will fail for those inside and possible
blocked.
The `HTTP_PROXY` environment variable seems to be a standard,
and therefore Excon is built to use it when available.
Excon is used by docker-api as well as fog, it might be used
by other components and there might be other parts that use
another HTTP client like RestClient, which also respects the
variable.
This means at the moment with that environment variable set
some requests would already rely on it.
In any case this should be in mentioned in the manual to be
aware of, also because some operating systems set this globally.
The question is should we make an afford to ensure deployment
behind a HTTP proxy on a system with HTTP blocked works
without issues and provide a way to configure it properly?
I've tested Foreman with HTTP blocked and `HTTP_PROXY` set,
but in a very basic setup, with the only external requests
being to Docker registries outside and squid configured to
just pass requests through regardless there to.
It didn't show any apparent issue, but there are for sure
issues with a more robust configured HTTP proxy.
This raises another question: How common is a setup where
external resources requiring HTTP are used with Foreman behind
a HTTP proxy?
Comments?
All the best,
Sebastian
[1] https://github.com/theforeman/foreman-docker/pull/189
<https://github.com/theforeman/foreman-docker/pull/189>
The biggest part of this all is actually ensuring that all requests made by
the application are actually going through the HTTP proxy.
As a solution to that, I was thinking of starting a plugin that configures
the proxy for HTTP libraries (Net::HTTP, Excon, RestClient, etc.) used.
At first it would only make sure that we have all requests covered.
As a nicety it could also have a debug mode to show outgoing requests via
something like httplog[1].
The HTTP proxy would at first just be one global setting, but later on by
extending the used libraries' underlying request methods it could
also allow for dynamically choosing the appropriate proxy per request.
This then can be used by the foreman_http_proxies[2] plugin.
[1] https://github.com/trusche/httplog
[2] https://github.com/jlsherrill/foreman_http_proxies
···
On Thursday, May 25, 2017 at 11:58:12 AM UTC+2, Marek Hulán wrote:
>
> sorry for typo, it should have been:
>
> AFAIK we don't *have* the solution *implemented* atm.
>
> --
> Marek
>
> On čtvrtek 25. května 2017 11:49:24 CEST Marek Hulán wrote:
> > Sorry for being late to the party, sending my 2c:
> >
> > I agree with having more complicated solution where users could have
> > separate proxies per service is good long-term goal. AFAIK we don't the
> > solution atm. Therefore I think introducing support for single, global
> > proxy sounds as improvement already to what we have now (nothing).
> What's
> > good on this, migrating to specific proxies should be easy, the RFC
> > explicitly[1] mentions it. The global proxy can have granular rules of
> what
> > communication should be passed through untouched and what should be sent
> > through maybe other proxies. Another advantage I see is that the global
> > proxy offloads the configuration from Foreman/Katello, which does not
> > really belongs into our domain.
> >
> > Later, when the RFC is implemented via foreman_http_proxies plugin, I'm
> > happy to stop using global proxy and improve plugins to use
> > foreman_http_proxies if it makes sense. It will take some time before
> > everyone adopts it. But meanwhile we'd still have the option to let user
> > configure their master proxy according to their needs.
> >
> > [1] https://github.com/theforeman/rfcs/pull/18/
> > files#diff-12584a6580dac145ae55c2b5d67088dfR45
> >
> > --
> > Marek
> >
> > On středa 17. května 2017 14:22:28 CEST Justin Sherrill wrote:
> > > On 05/17/2017 07:57 AM, Tom McKay wrote:
> > > > After reading the RFC I think that more robust and adaptable
> solution
> > > > would be better. A single env var is not going to cover the needs of
> > > > all the scenarios. A simple example may be accessing both
> > > > registry.access.redhat.com
> > > > (through proxy) and myopenshift:5000 (no proxy).
> > > >
> > > > As @jlsherrill noted on the PR, the temporary solution for the
> > > > foreman-docker plugin is alright for the moment.
> > >
> > > I'd like to echo what tom said, we've had many users that want to
> access
> > > content externally through a proxy and internally (where the proxy is
> > > not controlled by them and does not properly proxy internal requests).
> > > Its happened enough for me to say that a simple solution is not good
> > > enough long term.
> > >
> > > > On Wed, May 17, 2017 at 3:08 AM, Sebastian Gräßl > > > > > > > > <seba...@validcode.me >> wrote:
> > > > There was some feedback regarding this on the PR[1] mentioned in
> > > > the beginning.
> > > > There is already a RFC[2] regarding this and a plugin[3] to
> > > > implement the solution proposed in the RFC.
> > > >
> > > > The solution proposed by jlsherrill allows to add multiple
> > > > HTTP-proxies in Foreman and use these in plugins and allow to
> > > > configure what HTTP-proxy should be used for what requests.
> > > > So far the plugin only adds the ability to add HTTP proxies and
> > > > misses a essential part, which is applying the HTTP proxies to
> > > > requests.
> > > >
> > > > While looking at how other applications handle this and also
> > > > considering typical HTTP proxy configurations, it feels that
> such
> > > > a solution would make it rather complex in practice to apply.
> > > > Configuring rules for requests or just ensuring the proper
> request
> > > > is using the right HTTP proxy is better configurable in the HTTP
> > > > proxy itself.
> > > >
> > > > I believe a very bold simple solution would be the better, which
> > > > in my opinion would be to ensure all parts respect a HTTP proxy
> > > > configuration and have good documentation around it to advice on
> > > > how to configure the HTTP proxy correctly. Taken other
> > > > applications in the same area the HTTP_PROXY environment
> variable
> > > > seems to be the common standard to use.
> > > >
> > > > Please, I would love to hear more feedback on this and what are
> > > > common HTTP proxy setups.
> > > >
> > > > [1] https://github.com/theforeman/foreman-docker/pull/189
> > > >
> > > > [2] https://github.com/theforeman/rfcs/pull/18
> > > >
> > > > [3] https://github.com/jlsherrill/foreman_http_proxies
> > > >
> > > >
> > > > On Thursday, April 20, 2017 at 1:07:33 PM UTC+2, Sebastian Gräßl > > > > > > > > wrote:
> > > > Hej,
> > > >
> > > > at the moment there is a PR[1] open on foreman-docker to set
> a
> > > > HTTP proxy for requests to registries.
> > > > The PR allows to set a HTTP proxy on the HTTP client, in
> this
> > > > case deep down Excon, only for registry requests.
> > > >
> > > > A HTTP proxy won't be set on requests if a `HTTP_PROXY`
> > > > environment variable is available, since it is an unlikely
> > > > setup to have registry request routed over a different proxy
> > > > than other requests. However setting it via the environment
> > > > variable will allow requests to succeed to resources
> available
> > > > by the HTTP proxy, but will fail for those inside and
> possible
> > > > blocked.
> > > >
> > > > The `HTTP_PROXY` environment variable seems to be a
> standard,
> > > > and therefore Excon is built to use it when available.
> > > > Excon is used by docker-api as well as fog, it might be used
> > > > by other components and there might be other parts that use
> > > > another HTTP client like RestClient, which also respects the
> > > > variable.
> > > >
> > > > This means at the moment with that environment variable set
> > > > some requests would already rely on it.
> > > > In any case this should be in mentioned in the manual to be
> > > > aware of, also because some operating systems set this
> globally.
> > > >
> > > > The question is should we make an afford to ensure
> deployment
> > > > behind a HTTP proxy on a system with HTTP blocked works
> > > > without issues and provide a way to configure it properly?
> > > >
> > > > I've tested Foreman with HTTP blocked and `HTTP_PROXY` set,
> > > > but in a very basic setup, with the only external requests
> > > > being to Docker registries outside and squid configured to
> > > > just pass requests through regardless there to.
> > > >
> > > > It didn't show any apparent issue, but there are for sure
> > > > issues with a more robust configured HTTP proxy.
> > > > This raises another question: How common is a setup where
> > > > external resources requiring HTTP are used with Foreman
> behind
> > > > a HTTP proxy?
> > > >
> > > > Comments?
> > > >
> > > > All the best,
> > > > Sebastian
> > > >
> > > > [1] https://github.com/theforeman/foreman-docker/pull/189
> > > >
>
>
>
Why do you want to introduce it in a new plugin? I can see the global proxy
setting in core. When we get to the point when we want to have separate proxy
per library/plugin/communication then I think we should start using the
foreman_http_proxies plugin and enhance it if needed. From my experience,
starting a plugin brings a lot of extra maintenance effort, such as test
infrastructure, redmine setup, plugin manual write-up, hammer plugin from
scratch, support in installer. Why not simply add a new setting in Foreman?
···
--
Marek
On středa 21. června 2017 16:28:38 CEST Sebastian Gräßl wrote:
The biggest part of this all is actually ensuring that all requests made by
the application are actually going through the HTTP proxy.
As a solution to that, I was thinking of starting a plugin that configures
the proxy for HTTP libraries (Net::HTTP, Excon, RestClient, etc.) used.
At first it would only make sure that we have all requests covered.
As a nicety it could also have a debug mode to show outgoing requests via
something like httplog[1].
The HTTP proxy would at first just be one global setting, but later on by
extending the used libraries’ underlying request methods it could
also allow for dynamically choosing the appropriate proxy per request.
This then can be used by the foreman_http_proxies[2] plugin.
[1] https://github.com/trusche/httplog
[2] https://github.com/jlsherrill/foreman_http_proxies
On Thursday, May 25, 2017 at 11:58:12 AM UTC+2, Marek Hulán wrote:
sorry for typo, it should have been:
AFAIK we don’t have the solution implemented atm.
On čtvrtek 25. května 2017 11:49:24 CEST Marek Hulán wrote:
Sorry for being late to the party, sending my 2c:
I agree with having more complicated solution where users could have
separate proxies per service is good long-term goal. AFAIK we don’t the
solution atm. Therefore I think introducing support for single, global
proxy sounds as improvement already to what we have now (nothing).
What’s
good on this, migrating to specific proxies should be easy, the RFC
explicitly[1] mentions it. The global proxy can have granular rules of
what
communication should be passed through untouched and what should be sent
through maybe other proxies. Another advantage I see is that the global
proxy offloads the configuration from Foreman/Katello, which does not
really belongs into our domain.
Later, when the RFC is implemented via foreman_http_proxies plugin, I’m
happy to stop using global proxy and improve plugins to use
foreman_http_proxies if it makes sense. It will take some time before
everyone adopts it. But meanwhile we’d still have the option to let user
configure their master proxy according to their needs.
[1] https://github.com/theforeman/rfcs/pull/18/
files#diff-12584a6580dac145ae55c2b5d67088dfR45
On 05/17/2017 07:57 AM, Tom McKay wrote:
After reading the RFC I think that more robust and adaptable
solution
would be better. A single env var is not going to cover the needs of
all the scenarios. A simple example may be accessing both
registry.access.redhat.com http://registry.access.redhat.com
(through proxy) and myopenshift:5000 (no proxy).
As @jlsherrill noted on the PR, the temporary solution for the
foreman-docker plugin is alright for the moment.
I’d like to echo what tom said, we’ve had many users that want to
access
content externally through a proxy and internally (where the proxy is
not controlled by them and does not properly proxy internal requests).
Its happened enough for me to say that a simple solution is not good
enough long term.
On Wed, May 17, 2017 at 3:08 AM, Sebastian Gräßl > > > > > > > > > > <seba...@validcode.me <javascript:> <mailto:seba...@validcode.me > > > > <javascript:>>> wrote:
There was some feedback regarding this on the PR[1] mentioned in
the beginning.
There is already a RFC[2] regarding this and a plugin[3] to
implement the solution proposed in the RFC.
The solution proposed by jlsherrill allows to add multiple
HTTP-proxies in Foreman and use these in plugins and allow to
configure what HTTP-proxy should be used for what requests.
So far the plugin only adds the ability to add HTTP proxies and
misses a essential part, which is applying the HTTP proxies to
requests.
While looking at how other applications handle this and also
considering typical HTTP proxy configurations, it feels that
such
a solution would make it rather complex in practice to apply.
Configuring rules for requests or just ensuring the proper
request
is using the right HTTP proxy is better configurable in the HTTP
proxy itself.
I believe a very bold simple solution would be the better, which
in my opinion would be to ensure all parts respect a HTTP proxy
configuration and have good documentation around it to advice on
how to configure the HTTP proxy correctly. Taken other
applications in the same area the HTTP_PROXY environment
variable
seems to be the common standard to use.
Please, I would love to hear more feedback on this and what are
common HTTP proxy setups.
[1] https://github.com/theforeman/foreman-docker/pull/189
<https://github.com/theforeman/foreman-docker/pull/189>
[2] https://github.com/theforeman/rfcs/pull/18
<https://github.com/theforeman/rfcs/pull/18>
[3] https://github.com/jlsherrill/foreman_http_proxies
<https://github.com/jlsherrill/foreman_http_proxies>
On Thursday, April 20, 2017 at 1:07:33 PM UTC+2, Sebastian Gräßl > > > > > > > > > > wrote:
Hej,
at the moment there is a PR[1] open on foreman-docker to set
a
HTTP proxy for requests to registries.
The PR allows to set a HTTP proxy on the HTTP client, in
this
case deep down Excon, only for registry requests.
A HTTP proxy won't be set on requests if a `HTTP_PROXY`
environment variable is available, since it is an unlikely
setup to have registry request routed over a different proxy
than other requests. However setting it via the environment
variable will allow requests to succeed to resources
available
by the HTTP proxy, but will fail for those inside and
possible
blocked.
The `HTTP_PROXY` environment variable seems to be a
standard,
and therefore Excon is built to use it when available.
Excon is used by docker-api as well as fog, it might be used
by other components and there might be other parts that use
another HTTP client like RestClient, which also respects the
variable.
This means at the moment with that environment variable set
some requests would already rely on it.
In any case this should be in mentioned in the manual to be
aware of, also because some operating systems set this
globally.
The question is should we make an afford to ensure
deployment
behind a HTTP proxy on a system with HTTP blocked works
without issues and provide a way to configure it properly?
I've tested Foreman with HTTP blocked and `HTTP_PROXY` set,
but in a very basic setup, with the only external requests
being to Docker registries outside and squid configured to
just pass requests through regardless there to.
It didn't show any apparent issue, but there are for sure
issues with a more robust configured HTTP proxy.
This raises another question: How common is a setup where
external resources requiring HTTP are used with Foreman
behind
a HTTP proxy?
Comments?
All the best,
Sebastian
[1] https://github.com/theforeman/foreman-docker/pull/189
<https://github.com/theforeman/foreman-docker/pull/189>
I haven't thought as far as having it as an official plugin yet and not
considered all the necessary parts for that.
I can also add it into core.
My thoughts on preferring to have it as a plugin were that,
even though the code should be relatively small,
as it would probably only be initialiser code to set the HTTP proxy for
various http libraries,
it does have a big impact on the whole application and
having it as a plugin allows to disable all of it's doing by uninstalling
it.
I actually also briefly thought that it be a generic rubygem, but that
would have gone way too far. 
···
On Thursday, June 22, 2017 at 2:08:13 PM UTC+2, Marek Hulán wrote:
Why do you want to introduce it in a new plugin? I can see the global
proxy
setting in core. When we get to the point when we want to have separate
proxy
per library/plugin/communication then I think we should start using the
foreman_http_proxies plugin and enhance it if needed. From my experience,
starting a plugin brings a lot of extra maintenance effort, such as test
infrastructure, redmine setup, plugin manual write-up, hammer plugin from
scratch, support in installer. Why not simply add a new setting in
Foreman?
–
Marek
On středa 21. června 2017 16:28:38 CEST Sebastian Gräßl wrote:
The biggest part of this all is actually ensuring that all requests made
by
the application are actually going through the HTTP proxy.
As a solution to that, I was thinking of starting a plugin that
configures
the proxy for HTTP libraries (Net::HTTP, Excon, RestClient, etc.) used.
At first it would only make sure that we have all requests covered.
As a nicety it could also have a debug mode to show outgoing requests
via
something like httplog[1].
The HTTP proxy would at first just be one global setting, but later on
by
extending the used libraries’ underlying request methods it could
also allow for dynamically choosing the appropriate proxy per request.
This then can be used by the foreman_http_proxies[2] plugin.
[1] https://github.com/trusche/httplog
[2] https://github.com/jlsherrill/foreman_http_proxies
On Thursday, May 25, 2017 at 11:58:12 AM UTC+2, Marek Hulán wrote:
sorry for typo, it should have been:
AFAIK we don’t have the solution implemented atm.
On čtvrtek 25. května 2017 11:49:24 CEST Marek Hulán wrote:
Sorry for being late to the party, sending my 2c:
I agree with having more complicated solution where users could have
separate proxies per service is good long-term goal. AFAIK we don’t
the
solution atm. Therefore I think introducing support for single,
global
proxy sounds as improvement already to what we have now (nothing).
What’s
good on this, migrating to specific proxies should be easy, the RFC
explicitly[1] mentions it. The global proxy can have granular rules
of
what
communication should be passed through untouched and what should be
sent
through maybe other proxies. Another advantage I see is that the
global
proxy offloads the configuration from Foreman/Katello, which does
not
really belongs into our domain.
Later, when the RFC is implemented via foreman_http_proxies plugin,
I’m
happy to stop using global proxy and improve plugins to use
foreman_http_proxies if it makes sense. It will take some time
before
everyone adopts it. But meanwhile we’d still have the option to let
user
configure their master proxy according to their needs.
[1] https://github.com/theforeman/rfcs/pull/18/
files#diff-12584a6580dac145ae55c2b5d67088dfR45
On 05/17/2017 07:57 AM, Tom McKay wrote:
After reading the RFC I think that more robust and adaptable
solution
would be better. A single env var is not going to cover the
needs of
all the scenarios. A simple example may be accessing both
registry.access.redhat.com http://registry.access.redhat.com
(through proxy) and myopenshift:5000 (no proxy).
As @jlsherrill noted on the PR, the temporary solution for the
foreman-docker plugin is alright for the moment.
I’d like to echo what tom said, we’ve had many users that want to
access
content externally through a proxy and internally (where the proxy
is
not controlled by them and does not properly proxy internal
requests).
Its happened enough for me to say that a simple solution is not
good
enough long term.
On Wed, May 17, 2017 at 3:08 AM, Sebastian Gräßl > > > > > > > > > > > > <seba...@validcode.me <javascript:> <mailto:seba...@validcode.me > > > > > > <javascript:>>> wrote:
There was some feedback regarding this on the PR[1]
mentioned in
the beginning.
There is already a RFC[2] regarding this and a plugin[3] to
implement the solution proposed in the RFC.
The solution proposed by jlsherrill allows to add multiple
HTTP-proxies in Foreman and use these in plugins and allow
to
configure what HTTP-proxy should be used for what requests.
So far the plugin only adds the ability to add HTTP proxies
and
misses a essential part, which is applying the HTTP proxies
to
requests.
While looking at how other applications handle this and also
considering typical HTTP proxy configurations, it feels that
such
a solution would make it rather complex in practice to
apply.
Configuring rules for requests or just ensuring the proper
request
is using the right HTTP proxy is better configurable in the
HTTP
proxy itself.
I believe a very bold simple solution would be the better,
which
in my opinion would be to ensure all parts respect a HTTP
proxy
configuration and have good documentation around it to
advice on
how to configure the HTTP proxy correctly. Taken other
applications in the same area the HTTP_PROXY environment
variable
seems to be the common standard to use.
Please, I would love to hear more feedback on this and what
are
common HTTP proxy setups.
[1] https://github.com/theforeman/foreman-docker/pull/189
<https://github.com/theforeman/foreman-docker/pull/189>
[2] https://github.com/theforeman/rfcs/pull/18
<https://github.com/theforeman/rfcs/pull/18>
[3] https://github.com/jlsherrill/foreman_http_proxies
<https://github.com/jlsherrill/foreman_http_proxies>
On Thursday, April 20, 2017 at 1:07:33 PM UTC+2, Sebastian > Gräßl > > > > > > > > > > > > wrote:
Hej,
at the moment there is a PR[1] open on foreman-docker to
set
a
HTTP proxy for requests to registries.
The PR allows to set a HTTP proxy on the HTTP client, in
this
case deep down Excon, only for registry requests.
A HTTP proxy won't be set on requests if a `HTTP_PROXY`
environment variable is available, since it is an
unlikely
setup to have registry request routed over a different
proxy
than other requests. However setting it via the
environment
variable will allow requests to succeed to resources
available
by the HTTP proxy, but will fail for those inside and
possible
blocked.
The `HTTP_PROXY` environment variable seems to be a
standard,
and therefore Excon is built to use it when available.
Excon is used by docker-api as well as fog, it might be
used
by other components and there might be other parts that
use
another HTTP client like RestClient, which also respects
the
variable.
This means at the moment with that environment variable
set
some requests would already rely on it.
In any case this should be in mentioned in the manual to
be
aware of, also because some operating systems set this
globally.
The question is should we make an afford to ensure
deployment
behind a HTTP proxy on a system with HTTP blocked works
without issues and provide a way to configure it
properly?
I've tested Foreman with HTTP blocked and `HTTP_PROXY`
set,
but in a very basic setup, with the only external
requests
being to Docker registries outside and squid configured
to
just pass requests through regardless there to.
It didn't show any apparent issue, but there are for
sure
issues with a more robust configured HTTP proxy.
This raises another question: How common is a setup
where
external resources requiring HTTP are used with Foreman
behind
a HTTP proxy?
Comments?
All the best,
Sebastian
[1]
https://github.com/theforeman/foreman-docker/pull/189
<https://github.com/theforeman/foreman-docker/pull/189>
