Foreman proxy content not synchronising

Problem:

Foreman proxy content not synchronising,

Expected outcome:

Installed forman proxy content for the this link:

https://theforeman.org/plugins/katello/nightly/installation/smart_proxy.html

Foreman and Proxy versions:
Pulp node
Version
1.5.0
Pulp server version
2.21
Database connection
Messaging connection
Workers
10
SSH
Version
0.2.1
TFTP
Version
1.24.0
TFTP server
for-proxy.local.com
Templates
Version
1.24.0
Foreman and Proxy plugin versions:
Pulp node
Version
1.5.0
Pulp server version
2.21
Database connection
Messaging connection
Workers
10
SSH
Version
0.2.1
TFTP
Version
1.24.0
TFTP server

for-proxy.local.com
Templates
Version
1.24.0
Distribution and version:

Other relevant data:

Here’s the output from /var/log/messages

  https://foreman.local.com/pulp/repos/test/Testing_Centos7/CentOS7_Content_View/custom/CentOS_7_x86_64/Katello_3_14/repodata/8db1dd1784f858f45ecc30ee56f2f33d65348cb10c29564eccfcaf93c5430789-filelists.xml.gz.
Jan  2 00:49:22 for-proxy pulp: pulp.server.controllers.repository:INFO: [608aa285] Fully resyncing due to use of force_full in config
Jan  2 00:49:22 for-proxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [608aa285] Downloading metadata files.
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (2): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (3): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (4): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (5): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: pulp.server.controllers.repository:INFO: [0bf801e7] Fully resyncing due to use of force_full in config
Jan  2 00:49:22 for-proxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [0bf801e7] Downloading metadata files.
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (2): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (3): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (4): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (5): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.com/pulp/repos/test/Testing_Centos7/CentOS7_Content_View/custom/CentOS_7_x86_64/foreman-plugins-124/repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml.
Jan  2 00:49:22 for-proxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3ed5aacd] Parsing metadata.
Jan  2 00:49:22 for-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.com/pulp/repos/test/Testing_Centos7/CentOS7_Content_View/custom/CentOS_7_x86_64/Extras_x86_64/repodata/0784c5d3274cba83d7e6107965265f87504bad0d0c73c083649497436c4ab2be-filelists.xml.gz.
Jan  2 00:49:22 for-proxy pulp: pulp.server.controllers.repository:INFO: [3ed5aacd] Fully resyncing due to use of force_full in config
Jan  2 00:49:22 for-proxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3ed5aacd] Downloading metadata files.
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (2): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (3): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (4): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (5): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.com/pulp/repos/test/Testing_Centos7/CentOS7_Content_View/custom/CentOS_7_x86_64/foreman-plugins-124/repodata/7a7b0438a55a08de21727c1a4dfbfe479b9e6eeb17d7b6f5732d61e027c7b021-filelists.xml.gz.
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152) unsupported operand type(s) for -=: 'Retry' and 'int'
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152) Traceback (most recent call last):
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 274, in _fetch
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     **requests_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 498, in get
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     return self.request('GET', url, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     resp = self.send(prep, **send_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     r = adapter.send(request, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     timeout=timeout
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     _stacktrace=sys.exc_info()[2])
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 236, in increment
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     total -= 1
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152) TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440) unsupported operand type(s) for -=: 'Retry' and 'int'
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288) unsupported operand type(s) for -=: 'Retry' and 'int'
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440) Traceback (most recent call last):
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288) Traceback (most recent call last):
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 274, in _fetch
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 274, in _fetch
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     **requests_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     **requests_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 498, in get
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 498, in get
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     return self.request('GET', url, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     return self.request('GET', url, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     resp = self.send(prep, **send_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     resp = self.send(prep, **send_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     r = adapter.send(request, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     r = adapter.send(request, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     timeout=timeout
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     timeout=timeout
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     _stacktrace=sys.exc_info()[2])
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     _stacktrace=sys.exc_info()[2])
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 236, in increment
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 236, in increment
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     total -= 1
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288) TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'.
Katello::Errors::PulpError: RPM1004: Error retrieving metadata: Error 'unsupported operand type(s) for -=: 'Retry' and 'int'' 

https://foreman.local.com/pulp/repos/test/Testing_Centos7/CentOS7_Content_View/custom/CentOS_7_x86_64/foreman-plugins-124/repodata/7a7b0438a55a08de21727c1a4dfbfe479b9e6eeb17d7b6f5732d61e027c7b021-filelists.xml.gz

It keeps failing, and shows never sync status.

Please ignore this, sync is working when creating new repo, i set it to immediate instead of ondemand.
One thing dont know if it is a bug or misconfiguration which i doubt, the last sync always shows never, althought i can the repos are syncing.

I’m curious about this issue. Can you tell me what OS you are using what the python package versions of urllib3 and requests are?

OS CentOS Linux release 7.7.1908 (Core) , python-urllib3-1.10.2-7.el7.noarch.

I ran into the very same issue on my IPv6-only setup with Katello stable 3.14. I encountered this when doing initial sync of RHEL8 repositories:

After I re-initiiated sync the error disappeared.

I still need to know what version of python-requests is being used when the failure occurs. Can anyone share that with me?

Also, some searching around indicates that this problem might be ipv6 related:

I can confirm, this is IPv6 only setup. We have a document describing how to setup Katello with IPv6, this is work in progress.

FYI, I was able to reproduce this issue with python-requests 2.6.0-8.el7.7. There should be a new version of python requests out soon (2.6.0-11.el7) that should fix the issue. Alternatively, you can downgrade to 2.6.0-7.el7.7.

The bug is open at https://bugzilla.redhat.com/show_bug.cgi?id=1785666.

1 Like