Smart-proxy syncing of DRPMs failing

Katello server 3.3.1
Foreman Proxy 1.14.2

When syncing my katello content, certain repos fail to sync and therefore
never generate the repomd.xml, and then when servers try to use katello
content, they fail with a 404 and tell the user to disable the repos.

This is limited, in my environment, to CentOS 7 Extras, Updates, and Plus
repos. However, subscription-manager behaves in such a way that all
install/update tasks fail on these errors, instead of skipping the failing
repo.

From what I can tell, DRPMs are not correctly linked on the Katello server,
and so the sync process fails with 404 errors and "[foreman-tasks/action]
[E] PLP0000: Importer indicated a failed response
(Katello::Errors::PulpError)" reported on the Katello server.

The Smart Proxy logs the following error:
Mar 28 13:14:33 smart-proxy-01 pulp:
pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container
reported: {'downloads': {}, 'total_sources': 0} for base URL:
https://foreman-01.prod.mcs.som.mob.nuance.com/pulp/repos/nuance_mobility/Development/MCS_Mesos/custom/CentOS_7/Plus_x86_64/
Mar 28 13:14:33 smart-proxy-01 pulp:
pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 9 DRPMs.
Mar 28 13:14:33 smart-proxy-01 pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (1): foreman-01.prod.mcs.som.mob.nuance.com
Mar 28 13:14:33 smart-proxy-01 pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (2): foreman-01.prod.mcs.som.mob.nuance.com
Mar 28 13:14:33 smart-proxy-01 pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (3): foreman-01.prod.mcs.som.mob.nuance.com
Mar 28 13:14:33 smart-proxy-01 pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (4): foreman-01.prod.mcs.som.mob.nuance.com
Mar 28 13:14:33 smart-proxy-01 pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (5): foreman-01.prod.mcs.som.mob.nuance.com
Mar 28 13:14:33 smart-proxy-01 pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (6): foreman-01.prod.mcs.som.mob.nuance.com
Mar 28 13:14:33 smart-proxy-01 pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (7): foreman-01.prod.mcs.som.mob.nuance.com
Mar 28 13:14:33 smart-proxy-01 pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (8): foreman-01.prod.mcs.som.mob.nuance.com
Mar 28 13:14:33 smart-proxy-01 pulp:
requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS
connection (9): foreman-01.prod.mcs.som.mob.nuance.com
Mar 28 13:14:34 smart-proxy-01 pulp:
pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) 'error_code'
Mar 28 13:14:34 smart-proxy-01 pulp:
pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) Traceback (most
recent call last):
Mar 28 13:14:34 smart-proxy-01 pulp:
pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py",
line 232, in run
Mar 28 13:14:34 smart-proxy-01 pulp:
pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632)
self.update_content(metadata_files, url)
Mar 28 13:14:34 smart-proxy-01 pulp:
pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) File
"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py",
line 501, in update_content
Mar 28 13:14:34 smart-proxy-01 pulp:
pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) if
error[constants.ERROR_CODE] == constants.ERROR_KEY_ID_FILTER:
Mar 28 13:14:34 smart-proxy-01 pulp:
pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) KeyError:
'error_code'
Mar 28 13:14:34 smart-proxy-01 pulp: pulp.server.async.tasks:INFO: Task
failed : [4a88b023-5779-4b03-b1d1-b6c832d89d2b]
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
Task
pulp.server.managers.repo.sync.sync[4a88b023-5779-4b03-b1d1-b6c832d89d2b]
raised unexpected: PulpExecutionException('Importer indicated a failed
response',)
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
Traceback (most recent call last):
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in
trace_task
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
R = retval = fun(*args, **kwargs)
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line
488, in call
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
return super(Task, self).call(*args, **kwargs)
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line
103, in call
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
return super(PulpTask, self).call(*args, **kwargs)
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in
protected_call
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
return self.run(*args, **kwargs)
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
File
"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py",
line 810, in sync
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
raise pulp_exceptions.PulpExecutionException(_('Importer indicated a
failed response'))
Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632)
PulpExecutionException: Importer indicated a failed response

Corresponding 404s:
Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR:
Failed to find a catalog entry with path
"/var/lib/pulp/content/units/drpm/d5/29a4bfd392fd8b6a2bdeb0f85e403a4303cef05e521a8ad05065604473cd88/drpms/kernel-plus-doc-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.6.2.el7.centos.plus.noarch.drpm".
Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - -
[28/Mar/2017:13:15:00 +0000] "GET
/var/lib/pulp/content/units/drpm/d5/29a4bfd392fd8b6a2bdeb0f85e403a4303cef05e521a8ad05065604473cd88/drpms/kernel-plus-doc-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.6.2.el7.centos.plus.noarch.drpm
HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5
Linux/3.10.0-514.6.2.el7.x86_64"
Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR:
Failed to find a catalog entry with path
"/var/lib/pulp/content/units/drpm/0c/28310fe548a3368fc89b0c7127f8701a5ebc1bda6d0e4dc973ceb1ecb49059/drpms/python-perf-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.el7.centos.plus.x86_64.drpm".
Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - -
[28/Mar/2017:13:15:00 +0000] "GET
/var/lib/pulp/content/units/drpm/0c/28310fe548a3368fc89b0c7127f8701a5ebc1bda6d0e4dc973ceb1ecb49059/drpms/python-perf-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.el7.centos.plus.x86_64.drpm
HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5
Linux/3.10.0-514.6.2.el7.x86_64"
Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR:
Failed to find a catalog entry with path
"/var/lib/pulp/content/units/drpm/57/ea75483a4162b2f147c8a3b65a7358912b56b03b2a535614bb683be0c4654e/drpms/kernel-plus-abi-whitelists-3.10.0-327.28.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.noarch.drpm".
Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - -
[28/Mar/2017:13:15:00 +0000] "GET
/var/lib/pulp/content/units/drpm/57/ea75483a4162b2f147c8a3b65a7358912b56b03b2a535614bb683be0c4654e/drpms/kernel-plus-abi-whitelists-3.10.0-327.28.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.noarch.drpm
HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5
Linux/3.10.0-514.6.2.el7.x86_64"
Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR:
Failed to find a catalog entry with path
"/var/lib/pulp/content/units/drpm/b1/3e2047414a7492e14cc30182a80ae499c9dd289f59d34d4379d18e2cf8a399/drpms/perf-3.10.0-327.36.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.x86_64.drpm".
Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - -
[28/Mar/2017:13:15:00 +0000] "GET
/var/lib/pulp/content/units/drpm/b1/3e2047414a7492e14cc30182a80ae499c9dd289f59d34d4379d18e2cf8a399/drpms/perf-3.10.0-327.36.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.x86_64.drpm
HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5
Linux/3.10.0-514.6.2.el7.x86_64"
Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR:
Failed to find a catalog entry with path
"/var/lib/pulp/content/units/drpm/ed/ca7624263fa664662582eeae59b72946723675348b761ac17fc84bdf133515/drpms/kernel-plus-tools-libs-devel-3.10.0-327.36.3.el7.centos.plus_3.10.0-514.6.2.el7.centos.plus.x86_64.drpm".
Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - -
[28/Mar/2017:13:15:00 +0000] "GET
/var/lib/pulp/content/units/drpm/ed/ca7624263fa664662582eeae59b72946723675348b761ac17fc84bdf133515/drpms/kernel-plus-tools-libs-devel-3.10.0-327.36.3.el7.centos.plus_3.10.0-514.6.2.el7.centos.plus.x86_64.drpm
HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5
Linux/3.10.0-514.6.2.el7.x86_64"
Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR:
Failed to find a catalog entry with path
"/var/lib/pulp/content/units/drpm/5f/7a640ff1d0e560b76c79959e6acf05f0990f3b618acfe1c82221cc84687e2f/drpms/kernel-plus-tools-libs-3.10.0-327.36.1.el7.centos.plus_3.10.0-514.el7.centos.plus.x86_64.drpm".
Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - -
[28/Mar/2017:13:15:00 +0000] "GET
/var/lib/pulp/content/units/drpm/5f/7a640ff1d0e560b76c79959e6acf05f0990f3b618acfe1c82221cc84687e2f/drpms/kernel-plus-tools-libs-3.10.0-327.36.1.el7.centos.plus_3.10.0-514.el7.centos.plus.x86_64.drpm
HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5
Linux/3.10.0-514.6.2.el7.x86_64"
Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR:
Failed to find a catalog entry with path
"/var/lib/pulp/content/units/drpm/1f/62164ace61b1d3a93cb9ba6bc752f4671346051641641c208b996a5d573f9e/drpms/kernel-plus-doc-3.10.0-514.el7.centos.plus_3.10.0-514.2.2.el7.centos.plus.noarch.drpm".
Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - -
[28/Mar/2017:13:15:00 +0000] "GET
/var/lib/pulp/content/units/drpm/1f/62164ace61b1d3a93cb9ba6bc752f4671346051641641c208b996a5d573f9e/drpms/kernel-plus-doc-3.10.0-514.el7.centos.plus_3.10.0-514.2.2.el7.centos.plus.noarch.drpm
HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5
Linux/3.10.0-514.6.2.el7.x86_64"
Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR:
Failed to find a catalog entry with path
"/var/lib/pulp/content/units/drpm/3d/2864f9ed4d3f09349896192290f7aa661b308d764da8facc7cb6deb5fd7099/drpms/kernel-plus-tools-libs-3.10.0-327.28.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.x86_64.drpm".
Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - -
[28/Mar/2017:13:15:00 +0000] "GET
/var/lib/pulp/content/units/drpm/3d/2864f9ed4d3f09349896192290f7aa661b308d764da8facc7cb6deb5fd7099/drpms/kernel-plus-tools-libs-3.10.0-327.28.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.x86_64.drpm
HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5
Linux/3.10.0-514.6.2.el7.x86_64"
Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR:
Failed to find a catalog entry with path
"/var/lib/pulp/content/units/drpm/31/7797cc56856fc0e761dcaf5e2751b94b48ef2b009a31a42b70ff09d40242c5/drpms/kernel-plus-doc-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.el7.centos.plus.noarch.drpm".
Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - -
[28/Mar/2017:13:15:00 +0000] "GET
/var/lib/pulp/content/units/drpm/31/7797cc56856fc0e761dcaf5e2751b94b48ef2b009a31a42b70ff09d40242c5/drpms/kernel-plus-doc-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.el7.centos.plus.noarch.drpm
HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5
Linux/3.10.0-514.6.2.el7.x86_64"

If this is a problem specific to my environment/installation, I think I
remember seeing that there is/will be a tool to fully sync and repair these
repos - is that available for use now?

I fixed this by manually adding the drpms in the locations as specified in
the logs. Not ideal, but it worked. If that re-sync tool is in the works,
I'll sign up for it!

ยทยทยท On Tuesday, March 28, 2017 at 10:02:13 AM UTC-4, jpavel wrote: > > Katello server 3.3.1 > Foreman Proxy 1.14.2 > > When syncing my katello content, certain repos fail to sync and therefore > never generate the repomd.xml, and then when servers try to use katello > content, they fail with a 404 and tell the user to disable the repos. > > This is limited, in my environment, to CentOS 7 Extras, Updates, and Plus > repos. However, subscription-manager behaves in such a way that all > install/update tasks fail on these errors, instead of skipping the failing > repo. > > From what I can tell, DRPMs are not correctly linked on the Katello > server, and so the sync process fails with 404 errors and > "[foreman-tasks/action] [E] PLP0000: Importer indicated a failed response > (Katello::Errors::PulpError)" reported on the Katello server. > > The Smart Proxy logs the following error: > Mar 28 13:14:33 smart-proxy-01 pulp: > pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: The content container > reported: {'downloads': {}, 'total_sources': 0} for base URL: > https://foreman-01.prod.mcs.som.mob.nuance.com/pulp/repos/nuance_mobility/Development/MCS_Mesos/custom/CentOS_7/Plus_x86_64/ > Mar 28 13:14:33 smart-proxy-01 pulp: > pulp_rpm.plugins.importers.yum.sync:INFO: Downloading 9 DRPMs. > Mar 28 13:14:33 smart-proxy-01 pulp: > requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS > connection (1): foreman-01.prod.mcs.som.mob.nuance.com > Mar 28 13:14:33 smart-proxy-01 pulp: > requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS > connection (2): foreman-01.prod.mcs.som.mob.nuance.com > Mar 28 13:14:33 smart-proxy-01 pulp: > requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS > connection (3): foreman-01.prod.mcs.som.mob.nuance.com > Mar 28 13:14:33 smart-proxy-01 pulp: > requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS > connection (4): foreman-01.prod.mcs.som.mob.nuance.com > Mar 28 13:14:33 smart-proxy-01 pulp: > requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS > connection (5): foreman-01.prod.mcs.som.mob.nuance.com > Mar 28 13:14:33 smart-proxy-01 pulp: > requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS > connection (6): foreman-01.prod.mcs.som.mob.nuance.com > Mar 28 13:14:33 smart-proxy-01 pulp: > requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS > connection (7): foreman-01.prod.mcs.som.mob.nuance.com > Mar 28 13:14:33 smart-proxy-01 pulp: > requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS > connection (8): foreman-01.prod.mcs.som.mob.nuance.com > Mar 28 13:14:33 smart-proxy-01 pulp: > requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS > connection (9): foreman-01.prod.mcs.som.mob.nuance.com > Mar 28 13:14:34 smart-proxy-01 pulp: > pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) 'error_code' > Mar 28 13:14:34 smart-proxy-01 pulp: > pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) Traceback (most > recent call last): > Mar 28 13:14:34 smart-proxy-01 pulp: > pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) File > "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", > line 232, in run > Mar 28 13:14:34 smart-proxy-01 pulp: > pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) > self.update_content(metadata_files, url) > Mar 28 13:14:34 smart-proxy-01 pulp: > pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) File > "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", > line 501, in update_content > Mar 28 13:14:34 smart-proxy-01 pulp: > pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) if > error[constants.ERROR_CODE] == constants.ERROR_KEY_ID_FILTER: > Mar 28 13:14:34 smart-proxy-01 pulp: > pulp_rpm.plugins.importers.yum.sync:ERROR: (4409-85632) KeyError: > 'error_code' > Mar 28 13:14:34 smart-proxy-01 pulp: pulp.server.async.tasks:INFO: Task > failed : [4a88b023-5779-4b03-b1d1-b6c832d89d2b] > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > Task > pulp.server.managers.repo.sync.sync[4a88b023-5779-4b03-b1d1-b6c832d89d2b] > raised unexpected: PulpExecutionException('Importer indicated a failed > response',) > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > Traceback (most recent call last): > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 240, in > trace_task > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > R = retval = fun(*args, **kwargs) > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line > 488, in __call__ > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > return super(Task, self).__call__(*args, **kwargs) > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line > 103, in __call__ > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > return super(PulpTask, self).__call__(*args, **kwargs) > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 437, in > __protected_call__ > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > return self.run(*args, **kwargs) > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > File > "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", > line 810, in sync > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > raise pulp_exceptions.PulpExecutionException(_('Importer indicated a > failed response')) > Mar 28 13:14:34 smart-proxy-01 pulp: celery.worker.job:ERROR: (4283-85632) > PulpExecutionException: Importer indicated a failed response > > Corresponding 404s: > Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR: > Failed to find a catalog entry with path > "/var/lib/pulp/content/units/drpm/d5/29a4bfd392fd8b6a2bdeb0f85e403a4303cef05e521a8ad05065604473cd88/drpms/kernel-plus-doc-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.6.2.el7.centos.plus.noarch.drpm". > Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - - > [28/Mar/2017:13:15:00 +0000] "GET > /var/lib/pulp/content/units/drpm/d5/29a4bfd392fd8b6a2bdeb0f85e403a4303cef05e521a8ad05065604473cd88/drpms/kernel-plus-doc-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.6.2.el7.centos.plus.noarch.drpm > HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 > Linux/3.10.0-514.6.2.el7.x86_64" > Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR: > Failed to find a catalog entry with path > "/var/lib/pulp/content/units/drpm/0c/28310fe548a3368fc89b0c7127f8701a5ebc1bda6d0e4dc973ceb1ecb49059/drpms/python-perf-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.el7.centos.plus.x86_64.drpm". > Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - - > [28/Mar/2017:13:15:00 +0000] "GET > /var/lib/pulp/content/units/drpm/0c/28310fe548a3368fc89b0c7127f8701a5ebc1bda6d0e4dc973ceb1ecb49059/drpms/python-perf-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.el7.centos.plus.x86_64.drpm > HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 > Linux/3.10.0-514.6.2.el7.x86_64" > Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR: > Failed to find a catalog entry with path > "/var/lib/pulp/content/units/drpm/57/ea75483a4162b2f147c8a3b65a7358912b56b03b2a535614bb683be0c4654e/drpms/kernel-plus-abi-whitelists-3.10.0-327.28.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.noarch.drpm". > Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - - > [28/Mar/2017:13:15:00 +0000] "GET > /var/lib/pulp/content/units/drpm/57/ea75483a4162b2f147c8a3b65a7358912b56b03b2a535614bb683be0c4654e/drpms/kernel-plus-abi-whitelists-3.10.0-327.28.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.noarch.drpm > HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 > Linux/3.10.0-514.6.2.el7.x86_64" > Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR: > Failed to find a catalog entry with path > "/var/lib/pulp/content/units/drpm/b1/3e2047414a7492e14cc30182a80ae499c9dd289f59d34d4379d18e2cf8a399/drpms/perf-3.10.0-327.36.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.x86_64.drpm". > Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - - > [28/Mar/2017:13:15:00 +0000] "GET > /var/lib/pulp/content/units/drpm/b1/3e2047414a7492e14cc30182a80ae499c9dd289f59d34d4379d18e2cf8a399/drpms/perf-3.10.0-327.36.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.x86_64.drpm > HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 > Linux/3.10.0-514.6.2.el7.x86_64" > Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR: > Failed to find a catalog entry with path > "/var/lib/pulp/content/units/drpm/ed/ca7624263fa664662582eeae59b72946723675348b761ac17fc84bdf133515/drpms/kernel-plus-tools-libs-devel-3.10.0-327.36.3.el7.centos.plus_3.10.0-514.6.2.el7.centos.plus.x86_64.drpm". > Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - - > [28/Mar/2017:13:15:00 +0000] "GET > /var/lib/pulp/content/units/drpm/ed/ca7624263fa664662582eeae59b72946723675348b761ac17fc84bdf133515/drpms/kernel-plus-tools-libs-devel-3.10.0-327.36.3.el7.centos.plus_3.10.0-514.6.2.el7.centos.plus.x86_64.drpm > HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 > Linux/3.10.0-514.6.2.el7.x86_64" > Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR: > Failed to find a catalog entry with path > "/var/lib/pulp/content/units/drpm/5f/7a640ff1d0e560b76c79959e6acf05f0990f3b618acfe1c82221cc84687e2f/drpms/kernel-plus-tools-libs-3.10.0-327.36.1.el7.centos.plus_3.10.0-514.el7.centos.plus.x86_64.drpm". > Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - - > [28/Mar/2017:13:15:00 +0000] "GET > /var/lib/pulp/content/units/drpm/5f/7a640ff1d0e560b76c79959e6acf05f0990f3b618acfe1c82221cc84687e2f/drpms/kernel-plus-tools-libs-3.10.0-327.36.1.el7.centos.plus_3.10.0-514.el7.centos.plus.x86_64.drpm > HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 > Linux/3.10.0-514.6.2.el7.x86_64" > Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR: > Failed to find a catalog entry with path > "/var/lib/pulp/content/units/drpm/1f/62164ace61b1d3a93cb9ba6bc752f4671346051641641c208b996a5d573f9e/drpms/kernel-plus-doc-3.10.0-514.el7.centos.plus_3.10.0-514.2.2.el7.centos.plus.noarch.drpm". > Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - - > [28/Mar/2017:13:15:00 +0000] "GET > /var/lib/pulp/content/units/drpm/1f/62164ace61b1d3a93cb9ba6bc752f4671346051641641c208b996a5d573f9e/drpms/kernel-plus-doc-3.10.0-514.el7.centos.plus_3.10.0-514.2.2.el7.centos.plus.noarch.drpm > HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 > Linux/3.10.0-514.6.2.el7.x86_64" > Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR: > Failed to find a catalog entry with path > "/var/lib/pulp/content/units/drpm/3d/2864f9ed4d3f09349896192290f7aa661b308d764da8facc7cb6deb5fd7099/drpms/kernel-plus-tools-libs-3.10.0-327.28.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.x86_64.drpm". > Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - - > [28/Mar/2017:13:15:00 +0000] "GET > /var/lib/pulp/content/units/drpm/3d/2864f9ed4d3f09349896192290f7aa661b308d764da8facc7cb6deb5fd7099/drpms/kernel-plus-tools-libs-3.10.0-327.28.3.el7.centos.plus_3.10.0-514.6.1.el7.centos.plus.x86_64.drpm > HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 > Linux/3.10.0-514.6.2.el7.x86_64" > Mar 28 13:15:00 foreman-01 pulp_streamer: pulp.streamer.server:ERROR: > Failed to find a catalog entry with path > "/var/lib/pulp/content/units/drpm/31/7797cc56856fc0e761dcaf5e2751b94b48ef2b009a31a42b70ff09d40242c5/drpms/kernel-plus-doc-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.el7.centos.plus.noarch.drpm". > Mar 28 13:15:00 foreman-01 pulp_streamer: [-] 127.0.0.1 - - > [28/Mar/2017:13:15:00 +0000] "GET > /var/lib/pulp/content/units/drpm/31/7797cc56856fc0e761dcaf5e2751b94b48ef2b009a31a42b70ff09d40242c5/drpms/kernel-plus-doc-3.10.0-327.36.2.el7.centos.plus_3.10.0-514.el7.centos.plus.noarch.drpm > HTTP/1.1" 404 - "-" "python-requests/2.6.0 CPython/2.7.5 > Linux/3.10.0-514.6.2.el7.x86_64" > > > If this is a problem specific to my environment/installation, I think I > remember seeing that there is/will be a tool to fully sync and repair these > repos - is that available for use now? >