Foreman proxy content not synchronising

Brascon02 · January 2, 2020, 1:49am

Problem:

Foreman proxy content not synchronising,

Expected outcome:

Installed forman proxy content for the this link:

https://theforeman.org/plugins/katello/nightly/installation/smart_proxy.html

Foreman and Proxy versions:
Pulp node
Version
1.5.0
Pulp server version
2.21
Database connection
Messaging connection
Workers
10
SSH
Version
0.2.1
TFTP
Version
1.24.0
TFTP server
for-proxy.local.com
Templates
Version
1.24.0
Foreman and Proxy plugin versions:
Pulp node
Version
1.5.0
Pulp server version
2.21
Database connection
Messaging connection
Workers
10
SSH
Version
0.2.1
TFTP
Version
1.24.0
TFTP server

for-proxy.local.com
Templates
Version
1.24.0
Distribution and version:

Other relevant data:

Here’s the output from /var/log/messages

  https://foreman.local.com/pulp/repos/test/Testing_Centos7/CentOS7_Content_View/custom/CentOS_7_x86_64/Katello_3_14/repodata/8db1dd1784f858f45ecc30ee56f2f33d65348cb10c29564eccfcaf93c5430789-filelists.xml.gz.
Jan  2 00:49:22 for-proxy pulp: pulp.server.controllers.repository:INFO: [608aa285] Fully resyncing due to use of force_full in config
Jan  2 00:49:22 for-proxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [608aa285] Downloading metadata files.
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (2): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (3): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (4): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (5): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: pulp.server.controllers.repository:INFO: [0bf801e7] Fully resyncing due to use of force_full in config
Jan  2 00:49:22 for-proxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [0bf801e7] Downloading metadata files.
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (2): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (3): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (4): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (5): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.com/pulp/repos/test/Testing_Centos7/CentOS7_Content_View/custom/CentOS_7_x86_64/foreman-plugins-124/repodata/a27718cc28ec6d71432e0ef3e6da544b7f9d93f6bb7d0a55aacd592d03144b70-comps.xml.
Jan  2 00:49:22 for-proxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3ed5aacd] Parsing metadata.
Jan  2 00:49:22 for-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.com/pulp/repos/test/Testing_Centos7/CentOS7_Content_View/custom/CentOS_7_x86_64/Extras_x86_64/repodata/0784c5d3274cba83d7e6107965265f87504bad0d0c73c083649497436c4ab2be-filelists.xml.gz.
Jan  2 00:49:22 for-proxy pulp: pulp.server.controllers.repository:INFO: [3ed5aacd] Fully resyncing due to use of force_full in config
Jan  2 00:49:22 for-proxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [3ed5aacd] Downloading metadata files.
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (2): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (3): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (4): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (5): foreman.local.com
Jan  2 00:49:22 for-proxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.com/pulp/repos/test/Testing_Centos7/CentOS7_Content_View/custom/CentOS_7_x86_64/foreman-plugins-124/repodata/7a7b0438a55a08de21727c1a4dfbfe479b9e6eeb17d7b6f5732d61e027c7b021-filelists.xml.gz.
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152) unsupported operand type(s) for -=: 'Retry' and 'int'
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152) Traceback (most recent call last):
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 274, in _fetch
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     **requests_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 498, in get
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     return self.request('GET', url, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     resp = self.send(prep, **send_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     r = adapter.send(request, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     timeout=timeout
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     _stacktrace=sys.exc_info()[2])
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)   File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 236, in increment
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152)     total -= 1
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-97152) TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440) unsupported operand type(s) for -=: 'Retry' and 'int'
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288) unsupported operand type(s) for -=: 'Retry' and 'int'
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440) Traceback (most recent call last):
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288) Traceback (most recent call last):
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 274, in _fetch
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 274, in _fetch
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     **requests_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     **requests_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 498, in get
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 498, in get
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     return self.request('GET', url, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     return self.request('GET', url, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 486, in request
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     resp = self.send(prep, **send_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     resp = self.send(prep, **send_kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/requests/sessions.py", line 598, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     r = adapter.send(request, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     r = adapter.send(request, **kwargs)
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/requests/adapters.py", line 370, in send
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     timeout=timeout
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     timeout=timeout
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/urllib3/connectionpool.py", line 597, in urlopen
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)     _stacktrace=sys.exc_info()[2])
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     _stacktrace=sys.exc_info()[2])
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)   File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 236, in increment
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11673-01440)   File "/usr/lib/python2.7/site-packages/urllib3/util/retry.py", line 236, in increment
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288)     total -= 1
Jan  2 00:49:49 for-proxy pulp: nectar.downloaders.threaded:ERROR: (11692-72288) TypeError: unsupported operand type(s) for -=: 'Retry' and 'int'.
Katello::Errors::PulpError: RPM1004: Error retrieving metadata: Error 'unsupported operand type(s) for -=: 'Retry' and 'int''

https://foreman.local.com/pulp/repos/test/Testing_Centos7/CentOS7_Content_View/custom/CentOS_7_x86_64/foreman-plugins-124/repodata/7a7b0438a55a08de21727c1a4dfbfe479b9e6eeb17d7b6f5732d61e027c7b021-filelists.xml.gz

It keeps failing, and shows never sync status.

Brascon02 · January 5, 2020, 1:54pm

Please ignore this, sync is working when creating new repo, i set it to immediate instead of ondemand.
One thing dont know if it is a bug or misconfiguration which i doubt, the last sync always shows never, althought i can the repos are syncing.

daviddavis · January 6, 2020, 3:55pm

I’m curious about this issue. Can you tell me what OS you are using what the python package versions of urllib3 and requests are?

Brascon02 · January 7, 2020, 8:49pm

OS CentOS Linux release 7.7.1908 (Core) , python-urllib3-1.10.2-7.el7.noarch.

lzap · January 15, 2020, 11:46am

I ran into the very same issue on my IPv6-only setup with Katello stable 3.14. I encountered this when doing initial sync of RHEL8 repositories:

After I re-initiiated sync the error disappeared.

daviddavis · January 17, 2020, 3:49pm

I still need to know what version of python-requests is being used when the failure occurs. Can anyone share that with me?

Also, some searching around indicates that this problem might be ipv6 related:

https://github.com/pypa/pip/issues/4779

lzap · January 22, 2020, 10:16am

I can confirm, this is IPv6 only setup. We have a document describing how to setup Katello with IPv6, this is work in progress.

daviddavis · February 4, 2020, 8:20pm

FYI, I was able to reproduce this issue with python-requests 2.6.0-8.el7.7. There should be a new version of python requests out soon (2.6.0-11.el7) that should fix the issue. Alternatively, you can downgrade to 2.6.0-7.el7.7.

The bug is open at https://bugzilla.redhat.com/show_bug.cgi?id=1785666.