Problem:
After setting up the smart proxy and starting synchronizing two lifecycle environments I noticed an error to come up with two repos, CentOS 7 Updates and CentOS 8 AppStream.
It always failed with the dreaded PLP0000: Importer indicated a failed response error message.
The task shows nothing more to go on:
{"pulp_tasks"=>
[{"exception"=>nil,
"task_type"=>"pulp.server.managers.repo.sync.sync",
"_href"=>"/pulp/api/v2/tasks/8cb2b252-98e0-4c41-90cd-5dde4e1c51e4/",
"task_id"=>"8cb2b252-98e0-4c41-90cd-5dde4e1c51e4",
"tags"=>
["pulp:repository:1-CentOS_8-Testing-183839de-c7a8-4d04-8c94-607850033059",
"pulp:action:sync"],
"finish_time"=>"2020-11-22T09:07:18Z",
"_ns"=>"task_status",
"start_time"=>"2020-11-22T09:03:58Z",
"traceback"=>
"Traceback (most recent call last):\n" +
" File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" +
" R = retval = fun(*args, **kwargs)\n" +
" File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 688, in __call__\n" +
" return super(Task, self).__call__(*args, **kwargs)\n" +
" File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 110, in __call__\n" +
" return super(PulpTask, self).__call__(*args, **kwargs)\n" +
" File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" +
" return self.run(*args, **kwargs)\n" +
" File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 854, in sync\n" +
" raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))\n" +
"PulpExecutionException: Importer indicated a failed response\n",
"spawned_tasks"=>[],
"progress_report"=>
{"yum_importer"=>
{"content"=>
{"size_total"=>0,
"items_left"=>0,
"items_total"=>0,
"state"=>"FAILED",
"size_left"=>0,
"details"=>
{"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
"error"=>"'NoneType' object has no attribute 'findall'",
"error_details"=>[]},
"comps"=>{"state"=>"NOT_STARTED"},
"purge_duplicates"=>{"state"=>"NOT_STARTED"},
"distribution"=>
{"items_total"=>0,
"state"=>"NOT_STARTED",
"error_details"=>[],
"items_left"=>0},
"modules"=>{"state"=>"NOT_STARTED"},
"errata"=>{"state"=>"NOT_STARTED"},
"metadata"=>{"state"=>"FINISHED"}}},
"queue"=>"reserved_resource_worker-0@smartproxy.local.lan.dq2",
"state"=>"error",
"worker_name"=>"reserved_resource_worker-0@smartproxy.local.lan",
"result"=>nil,
"error"=>
{"code"=>"PLP0000",
"data"=>{},
"description"=>"Importer indicated a failed response",
"sub_errors"=>[]},
"_id"=>{"$oid"=>"5fba29594fde002c9416f792"},
"id"=>"5fba29594fde002c9416f792"}],
"poll_attempts"=>{"total"=>35, "failed"=>1}}
The /v/l/m on the smart proxy also does not reveal more:
Nov 22 11:03:58 smartproxy pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[8937cdef-cfe7-41c8-b10d-812e4c5c96fe]
Nov 22 11:03:58 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading metadata from https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/.
Nov 22 11:03:58 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:03:58 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/repomd.xml.
Nov 22 11:03:59 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Parsing metadata.
Nov 22 11:03:59 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading metadata from https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/.
Nov 22 11:03:59 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:03:59 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/repomd.xml.
Nov 22 11:04:00 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Parsing metadata.
Nov 22 11:04:00 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading metadata from https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/.
Nov 22 11:04:00 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:00 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/repomd.xml.
Nov 22 11:04:01 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Parsing metadata.
Nov 22 11:04:01 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading metadata from https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/.
Nov 22 11:04:01 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:01 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/repomd.xml.
Nov 22 11:04:02 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Parsing metadata.
Nov 22 11:04:02 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading metadata files.
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/c0cace4d6bd977ad29096c46467682b30b7ef65aff60fe7f7238096f9003d898-updateinfo.xml.gz.
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/667e850af80b8f7b69b39dc141e1e995780b7cb81a2d3086881e5252f68ecefd-modules.yaml.
Nov 22 11:04:03 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/df50a1de23dd68bd0f3e519adf7f50d69bc1ca7d6a77e45309adaac97c99a539-comps.xml.
Nov 22 11:04:03 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/6a30e89b5ccdabdd13371fb1bbea39d8a5ade5958205a9af051d0c42bcb8ec6e-filelists.xml.gz.
Nov 22 11:04:03 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/e1f1603a1930a50482866d070de8e82b4c5e0a5f5072f60f0823f54af8563089-primary.xml.gz.
Nov 22 11:04:05 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/08c0edb9ddee732a7cd5b4f40c27770e24dc378f6560761cb7fcc401f4909280-other.xml.gz.
Nov 22 11:04:06 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Generating metadata databases.
Nov 22 11:04:33 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Determining which units need to be downloaded.
Nov 22 11:06:34 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading 0 RPMs.
Nov 22 11:06:52 smartproxy pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: [8cb2b252] The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/
Nov 22 11:06:52 smartproxy pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [8cb2b252] Removing missing units.
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) 'NoneType' object has no attribute 'findall'
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) Traceback (most recent call last):
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 294, in run
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) self.update_content(metadata_files, url)
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 627, in update_content
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) purge.purge_unwanted_units(metadata_files, self.conduit, self.config, catalog)
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 46, in purge_unwanted_units
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) remove_missing_groups(metadata_files, conduit)
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 165, in remove_missing_groups
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) remote_named_tuples = get_remote_units(file_function, group.GROUP_TAG, process_func)
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 277, in get_remote_units
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) for unit in package_info_generator:
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 64, in package_list_generator
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) package_info = process_func(element)
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py", line 32, in process_group_element
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) packagelist.findall('packagereq'))
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) AttributeError: 'NoneType' object has no attribute 'findall'
Nov 22 11:07:18 smartproxy pulp: pulp.server.async.tasks:INFO: [8cb2b252] Task failed : [8cb2b252-98e0-4c41-90cd-5dde4e1c51e4]
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) Task pulp.server.managers.repo.sync.sync[8cb2b252-98e0-4c41-90cd-5dde4e1c51e4] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) Traceback (most recent call last):
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) R = retval = fun(*args, **kwargs)
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) return super(Task, self).__call__(*args, **kwargs)
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) return super(PulpTask, self).__call__(*args, **kwargs)
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) return self.run(*args, **kwargs)
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 854, in sync
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) PulpExecutionException: Importer indicated a failed response
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:INFO: [8937cdef] Task pulp.server.async.tasks._release_resource[8937cdef-cfe7-41c8-b10d-812e4c5c96fe] succeeded in 0.0027873479994s: None
I did many tests over the last 3 days, inc sync, full sync, I even added a new repository to the product and replaced the previous one with the new one, however, as soon as the smart proxy gets synced, the same error pops up.
Interestingly though, the failing step seems to be Removing missing units. It never failed anywhere else. I tried a mongodb repair, also to no avail.
Now I am lost and would need some assistance. What can I do to further troubleshoot or fix the issue?
Expected outcome:
Foreman and Proxy versions:
Foreman 2.2.1
Katello 3.17
Pulp 2
Foreman and Proxy plugin versions:
Foreman 2.2.1
Katello 3.17
Pulp 2
Distribution and version:
CentOS 7.9
Other relevant data: