Smart Proxy synchronize always fails at the same repository

Problem:
After setting up the smart proxy and starting synchronizing two lifecycle environments I noticed an error to come up with two repos, CentOS 7 Updates and CentOS 8 AppStream.
It always failed with the dreaded PLP0000: Importer indicated a failed response error message.

The task shows nothing more to go on:

{"pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.sync.sync",
    "_href"=>"/pulp/api/v2/tasks/8cb2b252-98e0-4c41-90cd-5dde4e1c51e4/",
    "task_id"=>"8cb2b252-98e0-4c41-90cd-5dde4e1c51e4",
    "tags"=>
     ["pulp:repository:1-CentOS_8-Testing-183839de-c7a8-4d04-8c94-607850033059",
      "pulp:action:sync"],
    "finish_time"=>"2020-11-22T09:07:18Z",
    "_ns"=>"task_status",
    "start_time"=>"2020-11-22T09:03:58Z",
    "traceback"=>
     "Traceback (most recent call last):\n" +
     "  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" +
     "    R = retval = fun(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 688, in __call__\n" +
     "    return super(Task, self).__call__(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 110, in __call__\n" +
     "    return super(PulpTask, self).__call__(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" +
     "    return self.run(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 854, in sync\n" +
     "    raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))\n" +
     "PulpExecutionException: Importer indicated a failed response\n",
    "spawned_tasks"=>[],
    "progress_report"=>
     {"yum_importer"=>
       {"content"=>
         {"size_total"=>0,
          "items_left"=>0,
          "items_total"=>0,
          "state"=>"FAILED",
          "size_left"=>0,
          "details"=>
           {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
          "error"=>"'NoneType' object has no attribute 'findall'",
          "error_details"=>[]},
        "comps"=>{"state"=>"NOT_STARTED"},
        "purge_duplicates"=>{"state"=>"NOT_STARTED"},
        "distribution"=>
         {"items_total"=>0,
          "state"=>"NOT_STARTED",
          "error_details"=>[],
          "items_left"=>0},
        "modules"=>{"state"=>"NOT_STARTED"},
        "errata"=>{"state"=>"NOT_STARTED"},
        "metadata"=>{"state"=>"FINISHED"}}},
    "queue"=>"reserved_resource_worker-0@smartproxy.local.lan.dq2",
    "state"=>"error",
    "worker_name"=>"reserved_resource_worker-0@smartproxy.local.lan",
    "result"=>nil,
    "error"=>
     {"code"=>"PLP0000",
      "data"=>{},
      "description"=>"Importer indicated a failed response",
      "sub_errors"=>[]},
    "_id"=>{"$oid"=>"5fba29594fde002c9416f792"},
    "id"=>"5fba29594fde002c9416f792"}],
 "poll_attempts"=>{"total"=>35, "failed"=>1}}

The /v/l/m on the smart proxy also does not reveal more:

Nov 22 11:03:58 smartproxy pulp: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[8937cdef-cfe7-41c8-b10d-812e4c5c96fe]
Nov 22 11:03:58 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading metadata from https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/.
Nov 22 11:03:58 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:03:58 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/repomd.xml.
Nov 22 11:03:59 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Parsing metadata.
Nov 22 11:03:59 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading metadata from https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/.
Nov 22 11:03:59 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:03:59 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/repomd.xml.
Nov 22 11:04:00 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Parsing metadata.
Nov 22 11:04:00 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading metadata from https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/.
Nov 22 11:04:00 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:00 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/repomd.xml.
Nov 22 11:04:01 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Parsing metadata.
Nov 22 11:04:01 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading metadata from https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/.
Nov 22 11:04:01 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:01 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/repomd.xml.
Nov 22 11:04:02 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Parsing metadata.
Nov 22 11:04:02 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading metadata files.
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/c0cace4d6bd977ad29096c46467682b30b7ef65aff60fe7f7238096f9003d898-updateinfo.xml.gz.
Nov 22 11:04:02 smartproxy pulp: urllib3.connectionpool:INFO: Starting new HTTPS connection (1): foreman.local.lan
Nov 22 11:04:02 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/667e850af80b8f7b69b39dc141e1e995780b7cb81a2d3086881e5252f68ecefd-modules.yaml.
Nov 22 11:04:03 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/df50a1de23dd68bd0f3e519adf7f50d69bc1ca7d6a77e45309adaac97c99a539-comps.xml.
Nov 22 11:04:03 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/6a30e89b5ccdabdd13371fb1bbea39d8a5ade5958205a9af051d0c42bcb8ec6e-filelists.xml.gz.
Nov 22 11:04:03 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/e1f1603a1930a50482866d070de8e82b4c5e0a5f5072f60f0823f54af8563089-primary.xml.gz.
Nov 22 11:04:05 smartproxy pulp: nectar.downloaders.threaded:INFO: Download succeeded: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/repodata/08c0edb9ddee732a7cd5b4f40c27770e24dc378f6560761cb7fcc401f4909280-other.xml.gz.
Nov 22 11:04:06 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Generating metadata databases.
Nov 22 11:04:33 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Determining which units need to be downloaded.
Nov 22 11:06:34 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [8cb2b252] Downloading 0 RPMs.
Nov 22 11:06:52 smartproxy pulp: pulp_rpm.plugins.importers.yum.repomd.alternate:INFO: [8cb2b252] The content container reported: {'downloads': {}, 'total_sources': 0} for base URL: https://foreman.local.lan/pulp/repos/myorg/Testing/CentOS_8/custom/CentOS_8/CentOS_8_App_Stream/
Nov 22 11:06:52 smartproxy pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [8cb2b252] Removing missing units.
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) 'NoneType' object has no attribute 'findall'
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) Traceback (most recent call last):
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 294, in run
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)     self.update_content(metadata_files, url)
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 627, in update_content
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)     purge.purge_unwanted_units(metadata_files, self.conduit, self.config, catalog)
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 46, in purge_unwanted_units
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)     remove_missing_groups(metadata_files, conduit)
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 165, in remove_missing_groups
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)     remote_named_tuples = get_remote_units(file_function, group.GROUP_TAG, process_func)
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 277, in get_remote_units
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)     for unit in package_info_generator:
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 64, in package_list_generator
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)     package_info = process_func(element)
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py", line 32, in process_group_element
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624)     packagelist.findall('packagereq'))
Nov 22 11:07:18 smartproxy pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [8cb2b252] (2116-42624) AttributeError: 'NoneType' object has no attribute 'findall'
Nov 22 11:07:18 smartproxy pulp: pulp.server.async.tasks:INFO: [8cb2b252] Task failed : [8cb2b252-98e0-4c41-90cd-5dde4e1c51e4]
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) Task pulp.server.managers.repo.sync.sync[8cb2b252-98e0-4c41-90cd-5dde4e1c51e4] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) Traceback (most recent call last):
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624)     R = retval = fun(*args, **kwargs)
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624)     return super(Task, self).__call__(*args, **kwargs)
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624)     return super(PulpTask, self).__call__(*args, **kwargs)
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624)     return self.run(*args, **kwargs)
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 854, in sync
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:ERROR: [8cb2b252] (2116-42624) PulpExecutionException: Importer indicated a failed response
Nov 22 11:07:18 smartproxy pulp: celery.app.trace:INFO: [8937cdef] Task pulp.server.async.tasks._release_resource[8937cdef-cfe7-41c8-b10d-812e4c5c96fe] succeeded in 0.0027873479994s: None

I did many tests over the last 3 days, inc sync, full sync, I even added a new repository to the product and replaced the previous one with the new one, however, as soon as the smart proxy gets synced, the same error pops up.

Interestingly though, the failing step seems to be Removing missing units. It never failed anywhere else. I tried a mongodb repair, also to no avail.

Now I am lost and would need some assistance. What can I do to further troubleshoot or fix the issue?

Expected outcome:

Foreman and Proxy versions:
Foreman 2.2.1
Katello 3.17
Pulp 2

Foreman and Proxy plugin versions:
Foreman 2.2.1
Katello 3.17
Pulp 2

Distribution and version:
CentOS 7.9

Other relevant data:

Iā€™ve also hit upon this issue, when attempting to sync content to my Smart Proxy, after adding the CentOS 8 AppStream repo to the Content View Iā€™m promoting to the Lifecycle Environment on the Smart Proxy. (Actually, I donā€™t even have to explicitly run a capsule content sync, as one gets done implicitly as part of the process of promoting the updated version of the Content View.)

These are the versions of the packages on the main Foreman+Katello server:

$ rpm -qa --qf '%{VERSION}\t%{RELEASE}\t%{NAME}\n' centos-release {foreman,katello}\* | sort | grep -v ca-consumer
1.0     1       katello-default-ca
1.0     1       katello-server-ca
1.7.5   1.el7   katello-client-bootstrap
2.1.3   1.el7   foreman
2.1.3   1.el7   foreman-cli
2.1.3   1.el7   foreman-debug
2.1.3   1.el7   foreman-dynflow-sidekiq
2.1.3   1.el7   foreman-installer
2.1.3   1.el7   foreman-installer-katello
2.1.3   1.el7   foreman-postgresql
2.1.3   1.el7   foreman-proxy
2.1.3   1.el7   foreman-release
2.1.3   1.el7   foreman-service
2.7.0   1.el7   katello-certs-tools
3.16.1  1.el7   katello
3.16.1  1.el7   katello-common
3.16.1  1.el7   katello-debug
3.16.1  1.el7   katello-repos
7       8.2003.0.el7.centos     centos-release

And these are the package versions on the Smart Proxy:

$ rpm -qa --qf '%{VERSION}\t%{RELEASE}\t%{NAME}\n' centos-release {foreman,katello}\* | sort | grep -v ca-consumer
1.0     1       katello-default-ca
1.0     1       katello-server-ca
1.7.5   1.el7   katello-client-bootstrap
2.1.4   1.el7   foreman-debug
2.1.4   1.el7   foreman-installer
2.1.4   1.el7   foreman-installer-katello
2.1.4   1.el7   foreman-proxy
2.1.4   1.el7   foreman-release
2.7.0   1.el7   katello-certs-tools
3.16.1  1.el7   foreman-proxy-content
3.16.1  1.el7   katello-common
3.16.1  1.el7   katello-debug
3.16.1  1.el7   katello-repos
7       3.el7   foreman-release-scl
7       8.2003.0.el7.centos     centos-release

Here is the configuration of the repo Iā€™m trying to sync:

# hammer repository info --id 3
ID:                 3
Name:               CentOS 8 AppStream
Label:              CentOS_8_AppStream
Organization:       American Museum of Natural History
Red Hat Repository: no
Content Type:       yum
Mirror on Sync:     yes
URL:                http://mirrorlist.centos.org/?release=8&arch=x86_64&repo=AppStream&infra=stock
Publish Via HTTP:   yes
Published At:       *URL SCRUBBED*
Relative Path:      Default_Organization/Library/custom/CentOS_8/CentOS_8_AppStream
Download Policy:    immediate
HTTP Proxy:         
    HTTP Proxy Policy: global_default_http_proxy
Product:            
    ID:   2
    Name: CentOS 8
GPG Key:            

Sync:               
    Status:         Success
    Last Sync Date: about 21 hours
Created:            2020/10/15 20:34:25
Updated:            2020/10/15 20:34:28
Content Counts:     
    Packages:       5367
    Source RPMS:    0
    Package Groups: 58
    Errata:         0
    Module Streams: 66

Iā€™m including more data, in case itā€™s useful. Hereā€™s the perspective from one of the clients of my Smart Proxy:

$ sudo yum module list
Updating Subscription Management repositories.
subscription-manager plugin disabled 6 system repositories with respect of configuration in /etc/dnf/plugins/subscription-manager.conf
CentOS 8 Base                                                                                   16 kB/s | 2.1 kB     00:00    
CentOS 8 EPEL                                                                                   15 kB/s | 2.4 kB     00:00    
CentOS 8 Extras                                                                                 13 kB/s | 2.1 kB     00:00    
CentOS 8 RPM Fusion nonfree                                                                     16 kB/s | 2.1 kB     00:00    
CentOS 8 RPM Fusion free                                                                        15 kB/s | 2.1 kB     00:00    
CentOS 8 Foreman 2.1                                                                            16 kB/s | 2.1 kB     00:00    
CentOS 8 Remi safe                                                                              16 kB/s | 2.1 kB     00:00    
CentOS 8 Wazuh                                                                                  17 kB/s | 2.1 kB     00:00    
CentOS 8 EPEL Modular                                                                           18 kB/s | 2.5 kB     00:00    
CentOS 8 PowerTools                                                                             15 kB/s | 2.5 kB     00:00    
CentOS 8 Remi modular                                                                           19 kB/s | 2.5 kB     00:00    
CentOS 8 AppStream                                                                             456  B/s | 224  B     00:00    
Errors during downloading metadata for repository 'Default_Organization_CentOS_8_CentOS_8_AppStream':
  - Status code: 404 for https://*SCRUBBED*/pulp/repos/Default_Organization/DMZ_Capsule/Capsule_test_1/custom/CentOS_8/CentOS_8_AppStream/repodata/repomd.xml (IP: *SCRUBBED*)
Error: Failed to download metadata for repo 'Default_Organization_CentOS_8_CentOS_8_AppStream': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried

This probably doesnā€™t demonstrate anything unexpected, since we already established that the AppStream repo isnā€™t getting synchronized successfully. But it may be useful to someone (or make it more likely that this is found via crawlers/indexers).

If I find any other useful bits in my troubleshooting, Iā€™ll update this post.

Its quite interesting to see, that you also experience that issue with the CentOS 8 AppStream repository. I have another one failing in the same way, CentOS 7 (base repo). All others are fine.

Currently I am trying to get rid of the AppStream repo and see if that helps. Creating a new repo inside the same CentOS 8 product with a different name didnā€™t help, so I assume its a more fundamental issue than just fluke.

Hmm, interesting. You have both the Foreman/Katello and the smart proxy installed on CentOS 8, I thought this is not supported yet?

Ronny,
Youā€™re correct, CentOS 8 is not supported yet. But Iā€™ve installed Foreman+Katello on CentOS 7. Have a closer look:

$ rpm -q centos-release
centos-release-7-8.2003.0.el7.centos.x86_64
$ cat /etc/centos-release
CentOS Linux release 7.8.2003 (Core)

In my previous output, I broke-out the VERSION and RELEASE query tags, to align things into neater columns, but take note that the VERSION tag has a value of ā€œ7ā€.

I havenā€™t yet gotten a chance to test this with the CentOS 7 repos. Iā€™ll keep an eye out for this when I do.

When I rolled-back the version of the Content View containing the AppStream repo, it resumed behaving in a functional manner again, with all of the other CentOS 8 repos that I included so-far.

Iā€™m not sure which way to go with it at this point. Perhaps Iā€™ll re-create the AppStream repo using an entirely different URL (e.g., by specifying one of the mirrorsā€™ URLs explicitly, rather than the ā€œmetaā€-URL that returns a 302 to one of the mirrors). Something to distinguish it from the original AppStream definition, in case Iā€™m running afoul of any de-duplication going on under the hood.

Thanks Paul, I was somehow only focused on the ā€œ8ā€ in there :slight_smile:

It will most likely not work, didnā€™t for me. Even if you fetch the repo from a different URL this new one will also not synchronized correctly as soon as you added to the CV. But please give it a try, maybe its something I missed.

The issue still persists in Katello 3.17.1.

I have the same problem in Katello 3.16.2, I have uploaded a foreman-debug in this post -> Katello smart proxy content sync error

Still working on this, will get an update soon. Sorry for the delay.

2 Likes

Thank you @cintrix84, let me know if you need any further debugging.

Hi all,

Not trying to put pressure on the issue or anything like that, but I believe Iā€™m experiencing this same challenge and would like to offer any information I can provide to help.

The only difference I can see (if Iā€™m properly understanding this thread thus far) however is CentOS8 or its AppStream repositories are not part of the equation in my situation.

Katello Server Packages:

# rpm -qa --qf '%{VERSION}\t%{RELEASE}\t%{NAME}\n' centos-release {foreman,katello}\* | sort | grep -v ca-consumer
1.0     1       katello-default-ca
1.0     1       katello-server-ca
1.7.5   1.el7   katello-client-bootstrap
2.1.4   1.el7   foreman
2.1.4   1.el7   foreman-cli
2.1.4   1.el7   foreman-debug
2.1.4   1.el7   foreman-dynflow-sidekiq
2.1.4   1.el7   foreman-installer
2.1.4   1.el7   foreman-installer-katello
2.1.4   1.el7   foreman-postgresql
2.1.4   1.el7   foreman-proxy
2.1.4   1.el7   foreman-release
2.1.4   1.el7   foreman-selinux
2.1.4   1.el7   foreman-service
2.7.0   1.el7   katello-certs-tools
3.16.2  1.el7   katello
3.16.2  1.el7   katello-common
3.16.2  1.el7   katello-debug
3.16.2  2.el7   katello-repos
3.4.0   1.el7   katello-selinux
3.5.4   1.el7   katello-agent
3.5.4   1.el7   katello-host-tools
3.5.4   1.el7   katello-host-tools-fact-plugin
7       9.2009.1.el7.centos     centos-release

SmartProxy Packages:

# rpm -qa --qf '%{VERSION}\t%{RELEASE}\t%{NAME}\n' centos-release {foreman,katello}\* | sort | grep -v ca-consumer
1.0     1       katello-default-ca
1.0     1       katello-server-ca
1.7.5   1.el7   katello-client-bootstrap
2.1.4   1.el7   foreman-debug
2.1.4   1.el7   foreman-installer
2.1.4   1.el7   foreman-installer-katello
2.1.4   1.el7   foreman-proxy
2.1.4   1.el7   foreman-release
2.7.0   1.el7   katello-certs-tools
3.16.2  1.el7   foreman-proxy-content
3.16.2  1.el7   katello-common
3.16.2  1.el7   katello-debug
3.16.2  2.el7   katello-repos
3.5.4   1.el7   katello-agent
3.5.4   1.el7   katello-host-tools
3.5.4   1.el7   katello-host-tools-fact-plugin
7       9.2009.1.el7.centos     centos-release

I actually just stood this instance up in my lab. I created a product with the CentOS7 ā€œOSā€, ā€œUpdatesā€, and ā€œExtrasā€ repositories:

# hammer product info --id 1 --organization "MyOrg"
ID:                1
Name:              CentOS 7 Linux for x86_64
Label:             CentOS_7_Linux_for_x86_64
Description:
Sync State (all):  Syncing Complete.
Sync State (last): Syncing Complete.
Sync Plan ID:
GPG:
    GPG Key ID:
    GPG Key:
Organization:      MyOrg
Content:
 1) Repo Name:    CentOS7-Extras-x86_64
    URL:          /custom/CentOS_7_Linux_for_x86_64/CentOS7-Extras-x86_64
    Content Type: yum
 2) Repo Name:    CentOS7-Foreman-Client-2.1-x86_64
    URL:          /custom/CentOS_7_Linux_for_x86_64/CentOS7-Foreman-Client-2_1-x86_64
    Content Type: yum
 3) Repo Name:    CentOS7-OS-x86_64
    URL:          /custom/CentOS_7_Linux_for_x86_64/CentOS7-OS-x86_64
    Content Type: yum
 4) Repo Name:    CentOS7-Updates-x86_64
    URL:          /custom/CentOS_7_Linux_for_x86_64/CentOS7-Updates-x86_64
    Content Type: yum

And then created two content-views out of those repositories. Published and promoted.

# hammer content-view info --id 3
ID:                           3
Name:                         Server
Label:                        Server
Composite:                    false
Description:
Content Host Count:           1
Force Puppet:                 false
Solve Dependencies:           false
Organization:                 MyOrg
Yum Repositories:
 1) ID:    4
    Name:  CentOS7-Updates-x86_64
    Label: CentOS7-Updates-x86_64
 2) ID:    1
    Name:  CentOS7-Extras-x86_64
    Label: CentOS7-Extras-x86_64
 3) ID:    2
    Name:  CentOS7-Foreman-Client-2.1-x86_64
    Label: CentOS7-Foreman-Client-2_1-x86_64
 4) ID:    3
    Name:  CentOS7-OS-x86_64
    Label: CentOS7-OS-x86_64
Container Image Repositories:

OSTree Repositories:

Puppet Modules:

Lifecycle Environments:
 1) ID:   2
    Name: Library
 2) ID:   3
    Name: DEV
Versions:
 1) ID:        3
    Version:   1.0
    Published: 2020/12/17 15:59:51
 2) ID:        5
    Version:   2.0
    Published: 2020/12/17 21:05:59
Components:

Activation Keys:
 1) centos_7_server_key_dev

I then assigned this content to my SmartProxy:

# hammer proxy content info --id 2
Lifecycle Environments:
 1) Name:          DEV
    Organization:  MyOrg
    Content Views:
     1) Name:           Server
        Composite:      no
        Last Published: 2020/12/17 21:05:59
        Content:
            Hosts:                 1
            Products:              1
            Yum repos:             4
            Container Image repos: 0
            Packages:              11312
            Package groups:        1
            Errata:                0
            Puppet modules:        0

I then try to synchronize this data, and of course this is where I hit the road block. Below is taken from production.log upon the failure:

2020-12-18T13:08:42 [I|app|02557b9c] Started POST "/katello/api/capsules/2-ods00298w10-example-net/content/sync" for 127.0.0.1 at 2020-12-18 13:08:42 +0000
2020-12-18T13:08:42 [I|app|02557b9c] Processing by Katello::Api::V2::CapsuleContentController#sync as HTML
2020-12-18T13:08:42 [I|app|02557b9c]   Parameters: {"id"=>"2-ods00298w10-example-net", "skip_metadata_check"=>false, "api_version"=>"v2", "capsule_content"=>{"id"=>"2-ods00298w10-example-net", "skip_metadata_check"=>false}}
2020-12-18T13:08:43 [I|bac|02557b9c] Task {label: Actions::Katello::CapsuleContent::Sync, id: 5e5e6043-b01c-4aab-af6c-95ec05012563, execution_plan_id: 432f95a7-8a48-4d2e-88cf-48b26b906538} state changed: planning
2020-12-18T13:08:44 [I|bac|] Task {label: Actions::Katello::CapsuleContent::Sync, id: 5e5e6043-b01c-4aab-af6c-95ec05012563, execution_plan_id: 432f95a7-8a48-4d2e-88cf-48b26b906538} state changed: planned
2020-12-18T13:08:44 [I|app|]   Rendering /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource
2020-12-18T13:08:44 [I|app|]   Rendered /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/views/katello/api/v2/common/async.json.rabl within katello/api/v2/layouts/resource (Duration: 51.4ms | Allocations: 13921)
2020-12-18T13:08:44 [I|app|] Completed 202 Accepted in 1404ms (Views: 51.4ms | ActiveRecord: 258.7ms | Allocations: 146479)
2020-12-18T13:08:44 [I|bac|] Task {label: Actions::Katello::CapsuleContent::Sync, id: 5e5e6043-b01c-4aab-af6c-95ec05012563, execution_plan_id: 432f95a7-8a48-4d2e-88cf-48b26b906538} state changed: running
2020-12-18T13:12:48 [E|bac|] PLP0000: Importer indicated a failed response (Katello::Errors::PulpError)
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
[ sidekiq ]
[ concurrent-ruby ]
2020-12-18T13:12:48 [E|bac|] PLP0000: Importer indicated a failed response (Katello::Errors::PulpError)
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.16.2/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.1/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-2.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
[ sidekiq ]
[ concurrent-ruby ]
2020-12-18T13:12:48 [I|bac|] Task {label: Actions::Katello::CapsuleContent::Sync, id: 5e5e6043-b01c-4aab-af6c-95ec05012563, execution_plan_id: 432f95a7-8a48-4d2e-88cf-48b26b906538} state changed: stopped  result: warning
2020-12-18T13:12:48 [I|bac|] Task {label: Actions::Katello::CapsuleContent::Sync, id: 5e5e6043-b01c-4aab-af6c-95ec05012563, execution_plan_id: 432f95a7-8a48-4d2e-88cf-48b26b906538} state changed: stopped  result: warning

And also from the SmartProxy syslog:

Dec 18 13:08:53 ods00298w10 pulp: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[63eaa0e3-e9c0-4031-b8cc-fc95af75afe5]
Dec 18 13:08:53 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [63eaa0e3] Downloading metadata from https://hqret-foreman.example.net/pulp/repos/MyORG/DEV/Store_Server/custom/CentOS_7_Linux_for_x86_64/CentOS7-OS-x86_64/.
Dec 18 13:08:55 ods00298w10 pulp: urllib3.connectionpool:INFO: [63eaa0e3] Starting new HTTPS connection (1): hqret-foreman.example.net
Dec 18 13:08:56 ods00298w10 pulp: nectar.downloaders.threaded:INFO: [63eaa0e3] Download succeeded: https://hqret-foreman.example.net/pulp/repos/MyORG/DEV/Store_Server/custom/CentOS_7_Linux_for_x86_64/CentOS7-OS-x86_64/.
Dec 18 13:08:56 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [63eaa0e3] Downloading metadata from https://hqret-foreman.example.net/pulp/repos/MyORG/DEV/Store_Server/custom/CentOS_7_Linux_for_x86_64/CentOS7-OS-x86_64/.
Dec 18 13:08:57 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [63eaa0e3] Parsing metadata.
Dec 18 13:08:57 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [63eaa0e3] Downloading metadata from https://hqret-foreman.example.net/pulp/repos/MyORG/DEV/Store_Server/custom/CentOS_7_Linux_for_x86_64/CentOS7-OS-x86_64/.
Dec 18 13:08:58 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [63eaa0e3] Parsing metadata.
Dec 18 13:08:58 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [63eaa0e3] Downloading metadata from https://hqret-foreman.example.net/pulp/repos/MyORG/DEV/Store_Server/custom/CentOS_7_Linux_for_x86_64/CentOS7-OS-x86_64/.
Dec 18 13:08:59 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [63eaa0e3] Parsing metadata.
Dec 18 13:08:59 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [63eaa0e3] Downloading metadata files.
Dec 18 13:09:03 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [63eaa0e3] Generating metadata databases.
Dec 18 13:09:41 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [63eaa0e3] Determining which units need to be downloaded.
Dec 18 13:11:56 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.purge:INFO: [63eaa0e3] Removing missing units.
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000) 'NoneType' object has no attribute 'findall'
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000) Traceback (most recent call last):
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 294, in run
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)     self.update_content(metadata_files, url)
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 627, in update_content
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)     purge.purge_unwanted_units(metadata_files, self.conduit, self.config, catalog)
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 46, in purge_unwanted_units
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)     remove_missing_groups(metadata_files, conduit)
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 165, in remove_missing_groups
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)     remote_named_tuples = get_remote_units(file_function, group.GROUP_TAG, process_func)
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 277, in get_remote_units
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)     for unit in package_info_generator:
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 64, in package_list_generator
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)     package_info = process_func(element)
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py", line 32, in process_group_element
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000)     packagelist.findall('packagereq'))
Dec 18 13:12:31 ods00298w10 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [63eaa0e3] (4172-32000) AttributeError: 'NoneType' object has no attribute 'findall'
Dec 18 13:12:31 ods00298w10 pulp: pulp.server.async.tasks:INFO: [63eaa0e3] Task failed : [63eaa0e3-e9c0-4031-b8cc-fc95af75afe5]
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000) Task pulp.server.managers.repo.sync.sync[63eaa0e3-e9c0-4031-b8cc-fc95af75afe5] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000) Traceback (most recent call last):
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000)     R = retval = fun(*args, **kwargs)
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000)     return super(Task, self).__call__(*args, **kwargs)
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000)     return super(PulpTask, self).__call__(*args, **kwargs)
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000)     return self.run(*args, **kwargs)
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 854, in sync
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Dec 18 13:12:31 ods00298w10 pulp: celery.app.trace:ERROR: [63eaa0e3] (4172-32000) PulpExecutionException: Importer indicated a failed response

It was really this SmartProxy log that made me think this is perhaps related to this reported issue.

Finally, overall health of system seems to be okay, according to hammer ping:

# hammer ping
database:
    Status:          ok
    Server Response: Duration: 0ms
candlepin:
    Status:          ok
    Server Response: Duration: 30ms
candlepin_events:
    Status:          ok
    message:         1 Processed, 0 Failed
    Server Response: Duration: 0ms
candlepin_auth:
    Status:          ok
    Server Response: Duration: 25ms
katello_events:
    Status:          ok
    message:         0 Processed, 0 Failed
    Server Response: Duration: 0ms
pulp:
    Status:          ok
    Server Response: Duration: 565ms
pulp_auth:
    Status:          ok
    Server Response: Duration: 71ms
foreman_tasks:
    Status:          ok
    Server Response: Duration: 3ms

Please let me know if anything else I can provide that may assist with the troubleshooting effort.

1 Like

Hi guys,

Sorry we had our holiday shutdown right after this issue got created. We are back in the office now, I will resume working on this.

3 Likes

Got the Katello box stood up and synced the same repos you did, going to get the capsule installed and sync over the content view that I made that is similar to yours and hope to replicate the issue.

Here is my content view:

Here are the repos I have synced:

2 Likes

Looks good, I have posted some stacktraces and extra info in this thread. Bug: Smart Proxies do not sync (Katello 3.15 through 3.18 RC2)

That should point you at some some files to look at and what to look for :slight_smile:

@Thulium-Drake

I was able to reproduce the issue, here is the error from the smart proxy logs:

Jan  5 17:05:14 capsule pulp: pulp.server.content.sources.event:ERROR: (11650-70432) error: error reading package header
Jan  5 17:10:05 capsule pulp: celery.beat:INFO: Scheduler: Sending due task download_deferred_content (pulp.server.controllers.repository.queue_download_deferred)
Jan  5 17:10:05 capsule rsyslogd: imjournal: 21955 messages lost due to rate-limiting
Jan  5 17:10:05 capsule pulp: celery.worker.strategy:INFO: Received task: pulp.server.controllers.repository.queue_download_deferred[cbdabb87-ab2a-4243-b363-501c690043ef]
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792) 'NoneType' object has no attribute 'findall'
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792) Traceback (most recent call last):
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 294, in run
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)     self.update_content(metadata_files, url)
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 627, in update_content
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)     purge.purge_unwanted_units(metadata_files, self.conduit, self.config, catalog)
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 46, in purge_unwanted_units
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)     remove_missing_groups(metadata_files, conduit)
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 165, in remove_missing_groups
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)     remote_named_tuples = get_remote_units(file_function, group.GROUP_TAG, process_func)
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 277, in get_remote_units
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)     for unit in package_info_generator:
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 64, in package_list_generator
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)     package_info = process_func(element)
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py", line 32, in process_group_element
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792)     packagelist.findall('packagereq'))
Jan  5 17:10:09 capsule pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [090a5d20] (11650-49792) AttributeError: 'NoneType' object has no attribute 'findall'
Jan  5 17:10:10 capsule pulp: pulp.server.async.tasks:INFO: [090a5d20] Task failed : [090a5d20-97fa-4a2c-8ba2-f729683684bc]
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792) Task pulp.server.managers.repo.sync.sync[090a5d20-97fa-4a2c-8ba2-f729683684bc] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792) Traceback (most recent call last):
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792)     R = retval = fun(*args, **kwargs)
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792)     return super(Task, self).__call__(*args, **kwargs)
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792)     return super(PulpTask, self).__call__(*args, **kwargs)
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792)     return self.run(*args, **kwargs)
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 854, in sync
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Jan  5 17:10:11 capsule pulp: celery.app.trace:ERROR: [090a5d20] (11650-49792) PulpExecutionException: Importer indicated a failed response

Here is a screenshot of the failed task when trying to sync the same repos you have in your content view:

Let me know if that looks the same as far as the task page goes.

If so I will reach out to the @Pulp team and see if they have any insights as to what is going on as I look from the Katello side.

Chris

2 Likes

I am very happy you could reproduce the issue. Just as a reminder, my error message is slightly different, it is in the phase ā€œRemoving missing unitsā€.

Thank you! This is exactly the message I receive :slight_smile: so itā€™s a reproducable problem, thatā€™s good!

I also wrote a hacky patch to workaround the issue: