Katello rc8, pulp streamer and authenticated proxy

Hi,

How to configure Katello, Pulp-Streamer and Squid for use a corporate
authenticated proxy ?

Thx

Nicolas

Nicolas,

Could you provide a little more detail on what you are trying to achieve?

Thanks,

John Mitsch
Red Hat Engineering
(860)-967-7285
irc: jomitsch

··· On Tue, Jun 7, 2016 at 10:43 AM, nd_dutyd2005 via Foreman users < foreman-users@googlegroups.com> wrote:

Hi,

How to configure Katello, Pulp-Streamer and Squid for use a corporate
authenticated proxy ?

Thx

Nicolas


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at https://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Oh , sorry.

Fresh install of katello rc8 on CentOS 7.

Configuration with corporate proxy : --katello-proxy-* parameters.

-> Create product & repos ok (download policy on-demand)
-> synchronization ok
-> subscription ok.

But, for yum update or yum install, there is an http 503 error, because
pulp_stream through squid, and squid not through by croporate proxy.

[Errno 14] HTTPS Error 503 - Service Unavailable-:–:-- ETA
Skipping requests to mirror.centos.org due to repeated connection failures:
('Connection aborted.', error(101, 'Network is unreachable')) -> blocked by
firewall, not through by corporate proxy

I try to modifiy ./pulp/server/plugins.conf.d/yum_importer.json by removing
proxy configuration, or to modify squid.conf by modifying cache_peer or
httpd conf in pulp_streamer.conf.
It's correct nothing.

Thx…

Nicolas

··· Le mardi 7 juin 2016 23:33:52 UTC+2, John Mitsch a écrit : > > Nicolas, > > Could you provide a little more detail on what you are trying to achieve? > > Thanks, > > John Mitsch > Red Hat Engineering > (860)-967-7285 > irc: jomitsch > > On Tue, Jun 7, 2016 at 10:43 AM, nd_dutyd2005 via Foreman users < > forema...@googlegroups.com > wrote: > >> Hi, >> >> How to configure Katello, Pulp-Streamer and Squid for use a corporate >> authenticated proxy ? >> >> Thx >> >> >> Nicolas >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Foreman users" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-user...@googlegroups.com . >> To post to this group, send email to forema...@googlegroups.com >> . >> Visit this group at https://groups.google.com/group/foreman-users. >> For more options, visit https://groups.google.com/d/optout. >> > >

In /etc/httpd/conf.d/pulp_streamer.conf configuration file:

  • adding ProxyRemote * "http://proxy_url:8080" -> Ok but it's an
    authenticated proxy -> [Errno 14] HTTPS Error 407 - Proxy Authentication
    Required ETA

It remains to be transmitted authorization to proxy…

··· Le mercredi 8 juin 2016 08:37:17 UTC+2, nd_dut...@yahoo.fr a écrit : > > Oh , sorry. > > Fresh install of katello rc8 on CentOS 7. > > Configuration with corporate proxy : --katello-proxy-* parameters. > > -> Create product & repos ok (download policy on-demand) > -> synchronization ok > -> subscription ok. > > But, for yum update or yum install, there is an http 503 error, because > pulp_stream through squid, and squid not through by croporate proxy. > > [Errno 14] HTTPS Error 503 - Service Unavailable-:--:-- ETA > Skipping requests to mirror.centos.org due to repeated connection > failures: ('Connection aborted.', error(101, 'Network is unreachable')) -> > blocked by firewall, not through by corporate proxy > > I try to modifiy ./pulp/server/plugins.conf.d/yum_importer.json by > removing proxy configuration, or to modify squid.conf by modifying > cache_peer or httpd conf in pulp_streamer.conf. > It's correct nothing. > > Thx.. > > Nicolas > > Le mardi 7 juin 2016 23:33:52 UTC+2, John Mitsch a écrit : >> >> Nicolas, >> >> Could you provide a little more detail on what you are trying to achieve? >> >> Thanks, >> >> John Mitsch >> Red Hat Engineering >> (860)-967-7285 >> irc: jomitsch >> >> On Tue, Jun 7, 2016 at 10:43 AM, nd_dutyd2005 via Foreman users < >> forema...@googlegroups.com> wrote: >> >>> Hi, >>> >>> How to configure Katello, Pulp-Streamer and Squid for use a corporate >>> authenticated proxy ? >>> >>> Thx >>> >>> >>> Nicolas >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Foreman users" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to foreman-user...@googlegroups.com. >>> To post to this group, send email to forema...@googlegroups.com. >>> Visit this group at https://groups.google.com/group/foreman-users. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >>

I progress…

Adding in /etc/httpd/conf.d/pulp_streamer.conf:
ProxyRemote * "http://proxy_url:8080"
RequestHeader set Proxy-Authorization "Basic <base64 username:password>"

yum update return now:
[Errno 14] HTTPS Error 301 - Moved Permanently

Nicolas,

Are you trying to sync repositories via on-demand or background? The
options would use pulp-streamer/squid. Can you also try to just grab them
using the immediate download policy? That should eliminate squid and
pulp-streamer from the equation.

You shouldn't really have to tweak the pulp_streamer or squid
configuration. As long as Satellite can fetch the repositories through your
corporate proxy, squid should work as well.

David

··· On Wednesday, June 8, 2016 at 9:33:38 AM UTC-4, nd_dut...@yahoo.fr wrote: > > I progress... > > Adding in /etc/httpd/conf.d/pulp_streamer.conf: > ProxyRemote * "http://proxy_url:8080" > RequestHeader set Proxy-Authorization "Basic " > > yum update return now: > [Errno 14] HTTPS Error 301 - Moved Permanently > >

Repositories are configured with on-demand policy.

But,now, i have an another problem :
An error occurred saving the Repository: There was an issue with the
backend service pulp_auth: 401 Unauthorized

··· Le mercredi 8 juin 2016 16:22:15 UTC+2, David a écrit : > > Nicolas, > > Are you trying to sync repositories via on-demand or background? The > options would use pulp-streamer/squid. Can you also try to just grab them > using the immediate download policy? That should eliminate squid and > pulp-streamer from the equation. > > You shouldn't really have to tweak the pulp_streamer or squid > configuration. As long as Satellite can fetch the repositories through your > corporate proxy, squid should work as well. > > David > > > On Wednesday, June 8, 2016 at 9:33:38 AM UTC-4, nd_dut...@yahoo.fr wrote: >> >> I progress... >> >> Adding in /etc/httpd/conf.d/pulp_streamer.conf: >> ProxyRemote * "http://proxy_url:8080" >> RequestHeader set Proxy-Authorization "Basic " >> >> yum update return now: >> [Errno 14] HTTPS Error 301 - Moved Permanently >> >>

With immediate policy, it's ok.

With on-demand or backgroup: [Errno 14] HTTPS Error 301 - Moved Permanently
in yum command.

··· Le mercredi 8 juin 2016 16:22:15 UTC+2, David a écrit : > > Nicolas, > > Are you trying to sync repositories via on-demand or background? The > options would use pulp-streamer/squid. Can you also try to just grab them > using the immediate download policy? That should eliminate squid and > pulp-streamer from the equation. > > You shouldn't really have to tweak the pulp_streamer or squid > configuration. As long as Satellite can fetch the repositories through your > corporate proxy, squid should work as well. > > David > > > On Wednesday, June 8, 2016 at 9:33:38 AM UTC-4, nd_dut...@yahoo.fr wrote: >> >> I progress... >> >> Adding in /etc/httpd/conf.d/pulp_streamer.conf: >> ProxyRemote * "http://proxy_url:8080" >> RequestHeader set Proxy-Authorization "Basic " >> >> yum update return now: >> [Errno 14] HTTPS Error 301 - Moved Permanently >> >>

I recapitulate:

yum install <package>

  • With basic configuration on-demand download policy

yum -> pulp-streamer -> external —> Error 503 in yum

  • WIth ProxyRemote * "http://proxy_url:8080" in
    /etc/httpd/conf.d/pulp_streamer.conf

yum -> pulp-streamer -> external —> Error 407 in yum (require proxy
authorization)

  • With RequestHeader set Proxy-Authorization "Basic <base64 user:pass>" and
    SetEnv proxy-chain-auth On in <Location /streamer/> block in
    /etc/httpd/conf.d/pulp_streamer.conf

yum -> pulp-streamer -> external —> Error 301 Redirect permanently in yum

Any idea to fix this little problem?

On-demand download policy is very intersting to download only required
packages.

··· Le jeudi 9 juin 2016 15:47:30 UTC+2, nd_dut...@yahoo.fr a écrit : > > With immediate policy, it's ok. > > With on-demand or backgroup: [Errno 14] HTTPS Error 301 - Moved > Permanently in yum command. > > Le mercredi 8 juin 2016 16:22:15 UTC+2, David a écrit : >> >> Nicolas, >> >> Are you trying to sync repositories via on-demand or background? The >> options would use pulp-streamer/squid. Can you also try to just grab them >> using the immediate download policy? That should eliminate squid and >> pulp-streamer from the equation. >> >> You shouldn't really have to tweak the pulp_streamer or squid >> configuration. As long as Satellite can fetch the repositories through your >> corporate proxy, squid should work as well. >> >> David >> >> >> On Wednesday, June 8, 2016 at 9:33:38 AM UTC-4, nd_dut...@yahoo.fr wrote: >>> >>> I progress... >>> >>> Adding in /etc/httpd/conf.d/pulp_streamer.conf: >>> ProxyRemote * "http://proxy_url:8080" >>> RequestHeader set Proxy-Authorization "Basic " >>> >>> yum update return now: >>> [Errno 14] HTTPS Error 301 - Moved Permanently >>> >>>

Well,

I found a solution.

Do not modify Apache configuration for pulp_streamer.conf

Add iptables rules:
iptables -t nat -N PASS
iptables -t nat -A PASS -j ACCEPT

iptables -t nat -A OUTPUT -d x.x.x.x/16 --proto tcp --dport 80 -j PASS
iptables -t nat -A OUTPUT -d x.x.x.x/16 --proto tcp --dport 443 -j PASS

iptables -t nat -A OUTPUT --proto tcp --dport 80 -j DNAT --to-destination
proxy_ip:3128
iptables -t nat -A OUTPUT --proto tcp --dport 443 -j DNAT --to-destination
proxy_ip:3128

But the external proxy reclaim authentication…
It's temporarely configure to be transparent proxy.

It's work with on-demand policy.

··· Le mercredi 15 juin 2016 14:36:27 UTC+2, nd_dut...@yahoo.fr a écrit : > > I recapitulate: > > yum install > > * With basic configuration on-demand download policy > > yum -> pulp-streamer -> external ---> Error 503 in yum > > * WIth ProxyRemote * "http://proxy_url:8080" in > /etc/httpd/conf.d/pulp_streamer.conf > > yum -> pulp-streamer -> external ---> Error 407 in yum (require proxy > authorization) > > * With RequestHeader set Proxy-Authorization "Basic " > and SetEnv proxy-chain-auth On in block in > /etc/httpd/conf.d/pulp_streamer.conf > > yum -> pulp-streamer -> external ---> Error 301 Redirect permanently in yum > > Any idea to fix this little problem? > > On-demand download policy is very intersting to download only required > packages. > > Le jeudi 9 juin 2016 15:47:30 UTC+2, nd_dut...@yahoo.fr a écrit : >> >> With immediate policy, it's ok. >> >> With on-demand or backgroup: [Errno 14] HTTPS Error 301 - Moved >> Permanently in yum command. >> >> Le mercredi 8 juin 2016 16:22:15 UTC+2, David a écrit : >>> >>> Nicolas, >>> >>> Are you trying to sync repositories via on-demand or background? The >>> options would use pulp-streamer/squid. Can you also try to just grab them >>> using the immediate download policy? That should eliminate squid and >>> pulp-streamer from the equation. >>> >>> You shouldn't really have to tweak the pulp_streamer or squid >>> configuration. As long as Satellite can fetch the repositories through your >>> corporate proxy, squid should work as well. >>> >>> David >>> >>> >>> On Wednesday, June 8, 2016 at 9:33:38 AM UTC-4, nd_dut...@yahoo.fr >>> wrote: >>>> >>>> I progress... >>>> >>>> Adding in /etc/httpd/conf.d/pulp_streamer.conf: >>>> ProxyRemote * "http://proxy_url:8080" >>>> RequestHeader set Proxy-Authorization "Basic " >>>> >>>> yum update return now: >>>> [Errno 14] HTTPS Error 301 - Moved Permanently >>>> >>>>

Hi,

The solution !

Pulp_Streamer -> Varnish -> netfilter rules -> Squid -> Authenticated
external proxy

This is not the most efficient solution because of the proxy stack.

··· Le vendredi 17 juin 2016 12:09:41 UTC+2, nd_dut...@yahoo.fr a écrit : > > Well, > > I found a solution. > > Do not modify Apache configuration for pulp_streamer.conf > > Add iptables rules: > iptables -t nat -N PASS > iptables -t nat -A PASS -j ACCEPT > > iptables -t nat -A OUTPUT -d x.x.x.x/16 --proto tcp --dport 80 -j PASS > iptables -t nat -A OUTPUT -d x.x.x.x/16 --proto tcp --dport 443 -j PASS > > iptables -t nat -A OUTPUT --proto tcp --dport 80 -j DNAT --to-destination > proxy_ip:3128 > iptables -t nat -A OUTPUT --proto tcp --dport 443 -j DNAT --to-destination > proxy_ip:3128 > > But the external proxy reclaim authentication... > It's temporarely configure to be transparent proxy. > > It's work with on-demand policy. > > > > Le mercredi 15 juin 2016 14:36:27 UTC+2, nd_dut...@yahoo.fr a écrit : >> >> I recapitulate: >> >> yum install >> >> * With basic configuration on-demand download policy >> >> yum -> pulp-streamer -> external ---> Error 503 in yum >> >> * WIth ProxyRemote * "http://proxy_url:8080" in >> /etc/httpd/conf.d/pulp_streamer.conf >> >> yum -> pulp-streamer -> external ---> Error 407 in yum (require proxy >> authorization) >> >> * With RequestHeader set Proxy-Authorization "Basic " >> and SetEnv proxy-chain-auth On in block in >> /etc/httpd/conf.d/pulp_streamer.conf >> >> yum -> pulp-streamer -> external ---> Error 301 Redirect permanently in >> yum >> >> Any idea to fix this little problem? >> >> On-demand download policy is very intersting to download only required >> packages. >> >> Le jeudi 9 juin 2016 15:47:30 UTC+2, nd_dut...@yahoo.fr a écrit : >>> >>> With immediate policy, it's ok. >>> >>> With on-demand or backgroup: [Errno 14] HTTPS Error 301 - Moved >>> Permanently in yum command. >>> >>> Le mercredi 8 juin 2016 16:22:15 UTC+2, David a écrit : >>>> >>>> Nicolas, >>>> >>>> Are you trying to sync repositories via on-demand or background? The >>>> options would use pulp-streamer/squid. Can you also try to just grab them >>>> using the immediate download policy? That should eliminate squid and >>>> pulp-streamer from the equation. >>>> >>>> You shouldn't really have to tweak the pulp_streamer or squid >>>> configuration. As long as Satellite can fetch the repositories through your >>>> corporate proxy, squid should work as well. >>>> >>>> David >>>> >>>> >>>> On Wednesday, June 8, 2016 at 9:33:38 AM UTC-4, nd_dut...@yahoo.fr >>>> wrote: >>>>> >>>>> I progress... >>>>> >>>>> Adding in /etc/httpd/conf.d/pulp_streamer.conf: >>>>> ProxyRemote * "http://proxy_url:8080" >>>>> RequestHeader set Proxy-Authorization "Basic >>>> username:password>" >>>>> >>>>> yum update return now: >>>>> [Errno 14] HTTPS Error 301 - Moved Permanently >>>>> >>>>>