Problem:
Trying to sync a new red hat repo from katello to content proxy does not work. Other redhat and non redhat repos appear to be syncing OK, but the content dir is not actually available on the capsule under the content url: /pulp/repos/Company/Library/content/. Library/ contains no RH repos:
Pulp Repository Content
Parent Directory
Centos7/
centos7_4-katello3_5-puppet5/
custom/
katello3_5/
puppet5/
listing
Initially I thought this was an issue with the RH repos that were originally synced last year, and went through the process of deleting the content views, and unchecking the RH repos in foreman, clearing orphaned files and then re-enabling the repos. This seems to be part of the problem with the content url not being listed - during sync operations the capsule says there are 0 RPM’s to download so they must be on disk.
In /var/lib/pulp/published/yum/master/yum_distributor on the capsule I can find many RH rpm’s, but the top level dir names don’t match the repo ID in foreman. As an example, the new repo I’ve added is the RHEL 7.6 kickstart repo, and I can find its id as a directory name under this path, but on the capsule I can not.
Expected outcome:
Foreman and Proxy versions:
1.16
Foreman and Proxy plugin versions:
Katello 3.5
Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)
logs
GUI sync task:
PLP0000: Importer indicated a failed response
RPM1004: Error retrieving metadata: NOT FOUND
RPM1004: Error retrieving metadata: NOT FOUND
RPM1004: Error retrieving metadata: NOT FOUND
PLP0000: Importer indicated a failed response
PLP0000: Importer indicated a failed response
capsule /var/log/messages:
2019-09-09T13:50:23.190455-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: [5f3cd458] The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: https://foreman001.domain.name/pulp/repos/Company/Library/content/dist/rhel/server/7/7.6/x86_64/kickstart/
2019-09-09T13:50:23.190851-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [5f3cd458] Removing missing units.
2019-09-09T13:50:41.311981-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [5f3cd458] Downloading additional units.
2019-09-09T13:50:41.316850-07:00 foreman-puppet001.domain.name pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman001.domain.name
2019-09-09T13:50:44.521718-07:00 foreman-puppet001.domain.name pulp: message repeated 3 times: [ requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman001.domain.name]
2019-09-09T13:50:44.524341-07:00 foreman-puppet001.domain.name pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (2): foreman001.domain.name
2019-09-09T13:50:44.526745-07:00 foreman-puppet001.domain.name pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): foreman001.domain.name
2019-09-09T13:50:44.527571-07:00 foreman-puppet001.domain.name pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): foreman001.domain.name
2019-09-09T13:50:44.529645-07:00 foreman-puppet001.domain.name pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (5): foreman001.domain.name
2019-09-09T13:50:48.952698-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-81632) ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))
2019-09-09T13:50:48.952785-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-81632) Traceback (most recent call last):
2019-09-09T13:50:48.952853-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-81632) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 290, in _fetch
2019-09-09T13:50:48.952969-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-81632) for chunk in chunks:
2019-09-09T13:50:48.953047-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-81632) File "/usr/lib/python2.7/site-packages/requests/models.py", line 661, in generate
2019-09-09T13:50:48.953121-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-81632) raise ContentDecodingError(e)
2019-09-09T13:50:48.953377-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-81632) ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))
2019-09-09T13:50:49.896619-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-71968) ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))
2019-09-09T13:50:49.896781-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-71968) Traceback (most recent call last):
2019-09-09T13:50:49.897074-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-71968) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 290, in _fetch
2019-09-09T13:50:49.897383-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-71968) for chunk in chunks:
2019-09-09T13:50:49.897486-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-71968) File "/usr/lib/python2.7/site-packages/requests/models.py", line 661, in generate
2019-09-09T13:50:49.897665-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-71968) raise ContentDecodingError(e)
2019-09-09T13:50:49.897750-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-71968) ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))
2019-09-09T13:50:50.180216-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-74336) ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))
2019-09-09T13:50:50.180404-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-74336) Traceback (most recent call last):
2019-09-09T13:50:50.180785-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-74336) File "/usr/lib/python2.7/site-packages/nectar/downloaders/threaded.py", line 290, in _fetch
2019-09-09T13:50:50.180864-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-74336) for chunk in chunks:
2019-09-09T13:50:50.180935-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-74336) File "/usr/lib/python2.7/site-packages/requests/models.py", line 661, in generate
2019-09-09T13:50:50.181104-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-74336) raise ContentDecodingError(e)
2019-09-09T13:50:50.181183-07:00 foreman-puppet001.domain.name pulp: nectar.downloaders.threaded:ERROR: (5610-74336) ContentDecodingError: ('Received response with content-encoding: gzip, but failed to decode it.', error('Error -3 while decompressing: incorrect header check',))
2019-09-09T13:50:52.534623-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.parse.treeinfo:ERROR: [5f3cd458] some distro file downloads failed
2019-09-09T13:50:52.758949-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368)
2019-09-09T13:50:52.759151-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368) Traceback (most recent call last):
2019-09-09T13:50:52.759236-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 277, in run
2019-09-09T13:50:52.759304-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368) dist_sync.run()
2019-09-09T13:50:52.759372-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 128, in run
2019-09-09T13:50:52.759436-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368) self._run(tmp_dir)
2019-09-09T13:50:52.759501-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 199, in _run
2019-09-09T13:50:52.759567-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368) for destination, location in downloaded:
2019-09-09T13:50:52.759632-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/parse/treeinfo.py", line 288, in download_files
2019-09-09T13:50:52.759695-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368) raise DownloadFailed()
2019-09-09T13:50:52.759759-07:00 foreman-puppet001.domain.name pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [5f3cd458] (5610-62368) DownloadFailed
2019-09-09T13:50:52.850641-07:00 foreman-puppet001.domain.name pulp: pulp.server.async.tasks:INFO: [5f3cd458] Task failed : [5f3cd458-8ee2-46ef-b74c-3957a3b4214f]
2019-09-09T13:50:52.861679-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) Task pulp.server.managers.repo.sync.sync[5f3cd458-8ee2-46ef-b74c-3957a3b4214f] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
2019-09-09T13:50:52.861861-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) Traceback (most recent call last):
2019-09-09T13:50:52.861905-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
2019-09-09T13:50:52.861975-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) R = retval = fun(*args, **kwargs)
2019-09-09T13:50:52.862043-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 527, in __call__
2019-09-09T13:50:52.862082-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) return super(Task, self).__call__(*args, **kwargs)
2019-09-09T13:50:52.862119-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 107, in __call__
2019-09-09T13:50:52.862156-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) return super(PulpTask, self).__call__(*args, **kwargs)
2019-09-09T13:50:52.862192-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
2019-09-09T13:50:52.862228-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) return self.run(*args, **kwargs)
2019-09-09T13:50:52.862265-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 827, in sync
2019-09-09T13:50:52.862301-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
2019-09-09T13:50:52.862337-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:ERROR: (1487-62368) PulpExecutionException: Importer indicated a failed response
2019-09-09T13:50:52.871113-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:INFO: Task pulp.server.async.tasks._release_resource[1de2a310-946c-4a25-a8f8-9bcc65b7513d] succeeded in 0.00519205699675s: None
2019-09-09T13:50:52.890620-07:00 foreman-puppet001.domain.name pulp: celery.worker.job:INFO: Task pulp.server.controllers.repository.download_deferred[51439673-52fd-4fa5-adab-b93c284ef15b] succeeded in 0.0158690789831s: None