Katello is bypassing the configured HTTP proxy

Hello Foreman community,

I’ve installed Foreman with Katello from scratch and I’m currently unable to synchronize any repository.

Problem:
Katello does not use the configured HTTP proxy for repository syncs at all.

Expected outcome:
Katello should use the configured HTTP proxy for repository syncs.

What I’ve done so far:
I’ve created a new proxy under Infrastructure > HTTP Proxies, where the proxy test (“Test Connection” button) itself did succeed, put the proxy as global default under Administer > Settings > Content > Default HTTP Proxy and double-checked my proxy settings for the repositories I want to synchronize. I’ve also tried doing this via hammer CLI and foreman-rake katello:update_default_http_proxy task - unfortunately without success. I could only verify the task to synchronize my repositories is trying to contact the upsteam URLs directly.

While browsing the forum, I found topics like this: How to sync yum repos via proxy
Promising, but unfortunately no solution anymore. Also, the --katello-proxy parameters for the foreman-installer are deprecated since around v1.24.

Test repository for yum content: https://packages.elastic.co/elasticsearch/2.x/centos

Any help would be much appreciated.

Foreman and Proxy versions:
Foreman: 2.1.2

Foreman and Proxy plugin versions:
Katello: 3.16

Distribution and version:
Oracle Linux 7.8

Other relevant data:
All I can do is to verify the proxy has not been used at all, except for the “Test Connection” button while creating or updating my HTTP proxy.

Basically, the following exception is raised, because the HTTP proxy is not being used:
Katello::Errors::Pulp3Error: Cannot connect to host packages.elastic.co:443 ssl:default [Network is unreachable]

I can provide more detailed information if needed.

Best regards

  1. Was this a new install from 3.16 or an upgrade from 3.15?

  2. You could also try to set a given repo to specifically use that proxy instead of use the global default on the repo details page (Content > products > click a product > click a repo). Change the http proxy option to use a specific http proxy, and choose it from the list.

I would be curious if that helps. Either way we can debug based on your answer to 1)

Hey Justin_Sherrill,

thanks a lot for helping me out.

  1. It was an entire new installation according to Foreman :: Manual
    and Foreman :: Plugin Manuals

  2. I’ve set the proxy explicitly now, instead of using the default HTTP proxy. Unfortunately, the problem still persists. Afterwards, I’ve updated the repository to no proxy and changed it back to my HTTP proxy - no success either. Also, I’ve created a new HTTP Proxy and another product and repository - same problem.

Can you run this command and provide the output:

sudo -u postgres psql pulpcore -c 'select url, proxy_url from core_remote'

I’m curious if your proxy url shows up in the output.

the pulp team identified a bug: https://pulp.plan.io/issues/7321 introduced in pulp_rpm 3.5. There is a patch available: https://github.com/pulp/pulp_rpm/pull/1837

if you want to try to apply it:

cd /usr/lib/python3.6/site-packages/pulp_rpm/
curl https://patch-diff.githubusercontent.com/raw/pulp/pulp_rpm/pull/1837.patch | patch -p2
systemctl restart pulpcore-worker@*
2 Likes

At first, let me provide the output you were asking for:

-bash-4.2$ psql pulpcore -c 'select url, proxy_url from core_remote'

                                             url                                              |         proxy_url
----------------------------------------------------------------------------------------------+----------------------------
 https://public-yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/                            | http://<proxy.example.com>
 https://public-yum.oracle.com/repo/OracleLinux/OL7/addons/x86_64/                            | http://<proxy.example.com>
 https://yum.oracle.com/repo/OracleLinux/OL8/appstream/x86_64/                                | http://<proxy.example.com>
 https://yum.oracle.com/repo/OracleLinux/OL8/addons/x86_64/                                   | http://<proxy.example.com>
 https://yum.oracle.com/repo/OracleLinux/OL8/baseos/latest/x86_64/                            | http://<proxy.example.com>
 https://packages.elastic.co/elasticsearch/2.x/centos                                         | http://<proxy.example.com>
(6 rows)

The proxy URL shows up for every single repository I’ve set up. Please note that I have replaced the original proxy URL from the output with http://<proxy.example.com>, of course :slight_smile:

Your second hint sounds very interesting, though I cannot apply the patch itself, since my current version of /usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py only has 52 lines, so it differs from https://github.com/pavelpicka/pulp_rpm/blob/c717196dab597f4a21ed71e3ce10097d14dc49f3/pulp_rpm/app/downloaders.py in more than just one change.
However, I could do the change manually, but I’m not sure if that’s the way to go.

Let me provide you some further information:

relevant pulp package

yum provides /usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py

python3-pulp-rpm-3.5.0-3.el7.noarch : RPM plugin for the Pulp Project
Repo        : @katello-pulpcore
Matched from:
Filename    : /usr/lib/python3.6/site-packages/pulp_rpm/app/downloaders.py

yum list installed | grep python3-pulp-rpm

python3-pulp-rpm.noarch               3.5.0-3.el7              @katello-pulpcore

katello-pulpcore repository

yum provides /etc/yum.repos.d/katello.repo

katello-repos-3.16.0-1.el7.noarch : Definition of yum repositories for Katello
Repo        : katello
Matched from:
Filename    : /etc/yum.repos.d/katello.repo

yum list installed | grep katello-repos

katello-repos.noarch                  3.16.0-1.el7             @/katello-repos-latest

/etc/yum.repos.d/katello.repo

[...]

[katello-pulpcore]
name=pulpcore: Fetch, Upload, Organize, and Distribute Software Packages.
baseurl=https://fedorapeople.org/groups/katello/releases/yum/3.16/pulpcore/el7/$basearch/
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-foreman
enabled=1
gpgcheck=1
module_hotfixes=1

[...]

I hope this helps. Thanks a lot for your support so far!

1 Like

After applying the patch manually (line 41 in my version of downloaders.py)

-        async with self.session.get(url) as response:
+        async with self.session.get(url, proxy=self.proxy, auth=self.auth) as response

and following your instructions by using systemctl restart pulpcore-worker@*, I was able to successfully synchronize my repositories again.

Thank you very much for your help!

3 Likes

As an update for this, this will be fixed in 3.16.1!

2 Likes

having the same issue as yours. manually applying the patch worked as well for me. Thanks dirgnirg and justin! :wink:

2 Likes

Thanks for letting us know @riceman08 and welcome to the community :smiley:

i’ve copied all class RpmDownloader(HttpDownloader) from downloaders.py. It works too.
So, when will be this issue fixed? on 3.17 version?
Thank you for advise.

Thanks heaps for pointing this one out Justin.

Applied the one line fix to downloaders.py and it worked a treat… until I ran out of space lol. Now I just need a crate of beer to convince the storage guys I need more disk.

2 Likes