Katello is bypassing the configured HTTP proxy

dirgnirg · September 4, 2020, 10:25am

Hello Foreman community,

I’ve installed Foreman with Katello from scratch and I’m currently unable to synchronize any repository.

Problem:
Katello does not use the configured HTTP proxy for repository syncs at all.

Expected outcome:
Katello should use the configured HTTP proxy for repository syncs.

What I’ve done so far:
I’ve created a new proxy under Infrastructure > HTTP Proxies, where the proxy test (“Test Connection” button) itself did succeed, put the proxy as global default under Administer > Settings > Content > Default HTTP Proxy and double-checked my proxy settings for the repositories I want to synchronize. I’ve also tried doing this via hammer CLI and foreman-rake katello:update_default_http_proxy task - unfortunately without success. I could only verify the task to synchronize my repositories is trying to contact the upsteam URLs directly.

While browsing the forum, I found topics like this: How to sync yum repos via proxy
Promising, but unfortunately no solution anymore. Also, the --katello-proxy parameters for the foreman-installer are deprecated since around v1.24.

Test repository for yum content: https://packages.elastic.co/elasticsearch/2.x/centos

Any help would be much appreciated.

Foreman and Proxy versions:
Foreman: 2.1.2

Foreman and Proxy plugin versions:
Katello: 3.16

Distribution and version:
Oracle Linux 7.8

Other relevant data:
All I can do is to verify the proxy has not been used at all, except for the “Test Connection” button while creating or updating my HTTP proxy.

Basically, the following exception is raised, because the HTTP proxy is not being used:
Katello::Errors::Pulp3Error: Cannot connect to host packages.elastic.co:443 ssl:default [Network is unreachable]

I can provide more detailed information if needed.

Best regards

Justin_Sherrill · September 4, 2020, 1:30pm

Was this a new install from 3.16 or an upgrade from 3.15?
You could also try to set a given repo to specifically use that proxy instead of use the global default on the repo details page (Content > products > click a product > click a repo). Change the http proxy option to use a specific http proxy, and choose it from the list.

I would be curious if that helps. Either way we can debug based on your answer to 1)

dirgnirg · September 4, 2020, 2:35pm

Hey Justin_Sherrill,

thanks a lot for helping me out.

It was an entire new installation according to Foreman :: Manual
and Foreman :: Plugin Manuals
I’ve set the proxy explicitly now, instead of using the default HTTP proxy. Unfortunately, the problem still persists. Afterwards, I’ve updated the repository to no proxy and changed it back to my HTTP proxy - no success either. Also, I’ve created a new HTTP Proxy and another product and repository - same problem.

Justin_Sherrill · September 4, 2020, 3:09pm

Can you run this command and provide the output:

sudo -u postgres psql pulpcore -c 'select url, proxy_url from core_remote'

I’m curious if your proxy url shows up in the output.

Justin_Sherrill · September 4, 2020, 6:21pm

the pulp team identified a bug: https://pulp.plan.io/issues/7321 introduced in pulp_rpm 3.5. There is a patch available: https://github.com/pulp/pulp_rpm/pull/1837

if you want to try to apply it:

cd /usr/lib/python3.6/site-packages/pulp_rpm/
curl https://patch-diff.githubusercontent.com/raw/pulp/pulp_rpm/pull/1837.patch | patch -p2
systemctl restart pulpcore-worker@*

dirgnirg · September 7, 2020, 7:15am

At first, let me provide the output you were asking for:

-bash-4.2$ psql pulpcore -c 'select url, proxy_url from core_remote'

                                             url                                              |         proxy_url
----------------------------------------------------------------------------------------------+----------------------------
 https://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/                            | http://<proxy.example.com>
 https://public-yum.oracle.com/repo/OracleLinux/OL7/addons/x86_64/                            | http://<proxy.example.com>
 https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/                                | http://<proxy.example.com>
 https://yum.oracle.com/repo/OracleLinux/OL8/addons/x86_64/                                   | http://<proxy.example.com>
 https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/                            | http://<proxy.example.com>
 https://packages.elastic.co/elasticsearch/2.x/centos                                         | http://<proxy.example.com>
(6 rows)

The proxy URL shows up for every single repository I’ve set up. Please note that I have replaced the original proxy URL from the output with http://<proxy.example.com>, of course

Your second hint sounds very interesting, though I cannot apply the patch itself, since my current version of /usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py only has 52 lines, so it differs from https://github.com/pavelpicka/pulp_rpm/blob/c717196dab597f4a21ed71e3ce10097d14dc49f3/pulp_rpm/app/downloaders.py in more than just one change.
However, I could do the change manually, but I’m not sure if that’s the way to go.

Let me provide you some further information:

relevant pulp package

yum provides /usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py

python3-pulp-rpm-3.5.0-3.el7.noarch : RPM plugin for the Pulp Project
Repo        : @katello-pulpcore
Matched from:
Filename    : /usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py

yum list installed | grep python3-pulp-rpm

python3-pulp-rpm.noarch               3.5.0-3.el7              @katello-pulpcore

katello-pulpcore repository

yum provides /etc/yum.repos.d/katello.repo

katello-repos-3.16.0-1.el7.noarch : Definition of yum repositories for Katello
Repo        : katello
Matched from:
Filename    : /etc/yum.repos.d/katello.repo

yum list installed | grep katello-repos

katello-repos.noarch                  3.16.0-1.el7             @/katello-repos-latest

/etc/yum.repos.d/katello.repo

[...]

[katello-pulpcore]
name=pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.
baseurl=https://fedorapeople.org/groups/katello/releases/yum/3.16/pulpcore/el7/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
enabled=1
gpgcheck=1
module_hotfixes=1

[...]

I hope this helps. Thanks a lot for your support so far!

dirgnirg · September 8, 2020, 6:16am

After applying the patch manually (line 41 in my version of downloaders.py)

-        async with self.session.get(url) as response:
+        async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response

and following your instructions by using systemctl restart pulpcore-worker@*, I was able to successfully synchronize my repositories again.

Thank you very much for your help!

Justin_Sherrill · September 11, 2020, 2:19pm

As an update for this, this will be fixed in 3.16.1!

riceman08 · September 30, 2020, 8:56am

having the same issue as yours. manually applying the patch worked as well for me. Thanks dirgnirg and justin!

mcorr · September 30, 2020, 3:13pm

Thanks for letting us know @riceman08 and welcome to the community

Andy.R · October 12, 2020, 8:03am

i’ve copied all class RpmDownloader(HttpDownloader) from downloaders.py. It works too.
So, when will be this issue fixed? on 3.17 version?
Thank you for advise.

andyfry · October 20, 2020, 11:36pm

Thanks heaps for pointing this one out Justin.

Applied the one line fix to downloaders.py and it worked a treat… until I ran out of space lol. Now I just need a crate of beer to convince the storage guys I need more disk.