Bug: Smart Proxies do not sync (Katello 3.15 through 3.18 RC2)

Thulium-Drake · December 11, 2020, 9:33pm

Problem: Smart Proxies do not finish syncing content succesfully (and do not work as proxy)

Hi Guys,

After the better part of this week trying to make it work, I’m lost. And after replying to a few older threads with some scattered info, I decided to group everything I found out so far in a single post

I have re-installed or upgraded my lab’s Foreman/Katello installation a dozen times by now. But there is just no scenario which I can find that does content syncing to proxies properly. I have ran the following scenario’s:

Composite content views, does not work, see the stacktrace below
Just a single non-composite content view, same error.
Re-installed instead of upgrading
Check with pulp-admin to see if I can get any more information, but it only throws PLP0000 with the python stacktrace as shown below in the journal.

The only situtation that does work (albeit unsurprisingly) is without any LCE assigned to the proxy. So the communication is there and works.

Things I noticed:

The ‘Content’ Tab of the proxy does show the published content views that are in the assigned LCE and the amount of packages that they contain
When the download_policy is set to ‘immediate’ the disk of the proxy is filled up with data
The errors seem (but I don’t know for sure!) to have “something” to do with yum groups?
It happens with both EL7 and EL8 repos (so not limited to AppStream or anything else)

Expected outcome: Content to be synced and proxies able to do host discovery, deployment etc.

Foreman and Proxy versions:
Foreman 2.0, 2.1, 2.2, 2.3. Either upgraded or freshly installed
Katello 3.15, 3.16, 3.17, 3.18. Either upgraded or freshly installed

Foreman and Proxy plugin versions:

Distribution and version:
CentOS 7
Other relevant data:

Python stacktrace

Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) 'NoneType' object has no attribute 'findall'
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) Traceback (most recent call last):
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 294, in run
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)     self.update_content(metadata_files, url)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py", line 627, in update_content
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)     purge.purge_unwanted_units(metadata_files, self.conduit, self.config, catalog)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 46, in purge_unwanted_units
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)     remove_missing_groups(metadata_files, conduit)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 165, in remove_missing_groups
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)     remote_named_tuples = get_remote_units(file_function, group.GROUP_TAG, process_func)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py", line 277, in get_remote_units
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)     for unit in package_info_generator:
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py", line 64, in package_list_generator
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)     package_info = process_func(element)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py", line 32, in process_group_element
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792)     packagelist.findall('packagereq'))
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp_rpm.plugins.importers.yum.sync:ERROR: [8a40d3d9] (23121-89792) AttributeError: 'NoneType' object has no attribute 'findall'
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: pulp.server.async.tasks:INFO: [8a40d3d9] Task failed : [8a40d3d9-a7f2-4694-8e5a-05bb317e6da9]
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) Task pulp.server.managers.repo.sync.sync[8a40d3d9-a7f2-4694-8e5a-05bb317e6da9] raised unexpected: PulpExecutionException('Importer indicated a failed response',)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) Traceback (most recent call last):
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792)     R = retval = fun(*args, **kwargs)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 688, in __call__
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792)     return super(Task, self).__call__(*args, **kwargs)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 110, in __call__
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792)     return super(PulpTask, self).__call__(*args, **kwargs)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792)     return self.run(*args, **kwargs)
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 854, in sync
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792)     raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))
Dec 11 16:14:38 smartproxy.lont.htm.lan pulp[23121]: celery.app.trace:ERROR: [8a40d3d9] (23121-89792) PulpExecutionException: Importer indicated a failed response

Output from Foreman Task

Actions::Pulp::Consumer::SyncCapsule

Input:

{"capsule_id"=>2,
 "repo_pulp_id"=>"1-CV_test-Library-1771db1b-36b8-4500-817e-234e7b9f8a4c",
 "repository_id"=>3,
 "sync_options"=>{"remove_missing"=>true},
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin",
 "current_request_id"=>nil,
 "current_timezone"=>"Europe/Amsterdam",
 "current_user_id"=>4,
 "current_organization_id"=>nil,
 "current_location_id"=>nil}

Output:

{"smart_proxy_history_id"=>3,
 "pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.sync.sync",
    "_href"=>"/pulp/api/v2/tasks/fc365a17-4ac8-4706-b5ac-da83522e8345/",
    "task_id"=>"fc365a17-4ac8-4706-b5ac-da83522e8345",
    "tags"=>
     ["pulp:repository:1-CV_test-Library-1771db1b-36b8-4500-817e-234e7b9f8a4c",
      "pulp:action:sync"],
    "finish_time"=>"2020-12-11T21:32:27Z",
    "_ns"=>"task_status",
    "start_time"=>"2020-12-11T21:29:29Z",
    "traceback"=>
     "Traceback (most recent call last):\n" +
     "  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" +
     "    R = retval = fun(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 688, in __call__\n" +
     "    return super(Task, self).__call__(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 110, in __call__\n" +
     "    return super(PulpTask, self).__call__(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" +
     "    return self.run(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 854, in sync\n" +
     "    raise pulp_exceptions.PulpExecutionException(_('Importer indicated a failed response'))\n" +
     "PulpExecutionException: Importer indicated a failed response\n",
    "spawned_tasks"=>[],
    "progress_report"=>
     {"yum_importer"=>
       {"content"=>
         {"size_total"=>0,
          "items_left"=>0,
          "items_total"=>0,
          "state"=>"FAILED",
          "size_left"=>0,
          "details"=>
           {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
          "error"=>"'NoneType' object has no attribute 'findall'",
          "error_details"=>[]},
        "comps"=>{"state"=>"NOT_STARTED"},
        "purge_duplicates"=>{"state"=>"NOT_STARTED"},
        "distribution"=>
         {"items_total"=>0,
          "state"=>"NOT_STARTED",
          "error_details"=>[],
          "items_left"=>0},
        "modules"=>{"state"=>"NOT_STARTED"},
        "errata"=>{"state"=>"NOT_STARTED"},
        "metadata"=>{"state"=>"FINISHED"}}},
    "queue"=>"reserved_resource_worker-0@smartproxy.lont.htm.lan.dq2",
    "state"=>"error",
    "worker_name"=>"reserved_resource_worker-0@smartproxy.lont.htm.lan",
    "result"=>nil,
    "error"=>
     {"code"=>"PLP0000",
      "data"=>{},
      "description"=>"Importer indicated a failed response",
      "sub_errors"=>[]},
    "_id"=>{"$oid"=>"5fd3e4b9ae22663ace405dd5"},
    "id"=>"5fd3e4b9ae22663ace405dd5"}],
 "poll_attempts"=>{"total"=>32, "failed"=>1}}

Exception:

Katello::Errors::PulpError: PLP0000: Importer indicated a failed response

Backtrace:

/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:121:in `block in external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:119:in `each'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:119:in `external_task='
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:100:in `poll_external_task_with_rescue'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/polling.rb:22:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/cancellable.rb:14:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/pulp/abstract_async_task.rb:45:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:571:in `block (3 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/record_smart_proxy_sync_history.rb:11:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/remote_action.rb:16:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/models/katello/concerns/user_extensions.rb:21:in `cp_config'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.0.rc2.1/app/lib/actions/middleware/remote_action.rb:16:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/execution_wrapper.rb:88:in `wrap'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:31:in `with_progress_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action/progress.rb:17:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:49:in `restore_current_request_id'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_request_id.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_timezone.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:44:in `restore_curent_user'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `block in run'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies'
/opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-3.0.2/app/lib/actions/middleware/keep_current_taxonomies.rb:15:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:27:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:19:in `pass'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware.rb:32:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/stack.rb:23:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/middleware/world.rb:31:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:570:in `block (2 levels) in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:569:in `block in execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `block in with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `catch'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:472:in `with_error_handling'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:564:in `execute_run'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:93:in `execute'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'

Thulium-Drake · December 14, 2020, 4:15pm

ping @katello anyone? I don’t know where to look for a solution to this issue, but as mentioned in my post, I did a bunch of installations in multiple versions and none of them currently work…

cintrix84 · December 14, 2020, 5:11pm

Hi @Thulium-Drake

Sorry for the delay, starting to look at this.

Thulium-Drake · December 14, 2020, 5:19pm

Awesome! Please let me know if there is something I need to test/dig up, I have a lab testbed with Katello 3.18 snapshotted and ready

rbremer · December 14, 2020, 6:27pm

Could be related to the other two issues regarding smart proxy sync. Can’t wait to find a solution for that.

Thulium-Drake · December 21, 2020, 11:59am

I may have found a workaround (WARNING: possible dirty hack detected). The problem seems to be with the Yum groups defined in comps.xml (I wasn’t able to find the file itself, but as most repos seem to have this problem, I decided ‘just’ to check Yum docs what it’s supposed to do)

And I edited the Python file (/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py) throwing the errors.

Replace

    packagelist = element.find('packagelist')
    conditional, default, mandatory, optional = _parse_packagelist(
        packagelist.findall('packagereq'))

with

    packagelist = element.find('packagelist')
    try:
        conditional, default, mandatory, optional = _parse_packagelist(
            packagelist.findall('packagereq'))
    except:
        conditional, default, mandatory, optional = [], [], [], []

This bypasses any faulty comps.xml files, but I DO NOT KNOW if and what this might break in the long term. I am still currently testing it.

I am able to sync a proxy now and just deployed my first VM using that proxy.

Please do NOT do this on any production system that is vital!

Thulium-Drake · December 22, 2020, 8:03am

So far so good, I have installed multiple systems via my ‘hacked’ proxy, they all work. With 1 exception, but I’m not entirely sure this is related to this fix.

Installing EL8 doesn’t run automatically because it can’t find wget and redhat-lsb-core. As these are both packages in AppStream, I think the fix made in Foreman server hasn’t been completely landed in the proxy component. But that’s just my guess

fresh-pie · December 22, 2020, 5:33pm

Hi!

Just wanted to say, I implemented this hack on my lab proxy and it did indeed allow me to workaround this issue. This is awesome, thank you! It may be a band-aid, but this allows me to continue my proof of concept and I appreciate it greatly.

I will continue to monitor this thread (and the others) and will be available for any additional testing.

Thulium-Drake · December 22, 2020, 5:38pm

Thanks for the feedback! Good to hear it’s working for you too. Maybe @cintrix84 can say something about the hackiness of this fix

Mamba · June 28, 2021, 5:44am

I am experiencing this issue as well.

As has already been stated the problem occurs when syncing a Foreman Server to a Smart Proxy.
It makes no difference if it is a Complete or Optimized Sync.

My test was done on a clean install of both the OS and the Foreman Software.

My scenario:
Foreman Server: CentOS Linux release 7.9.2009 (Core)
katello-3.18.3-1.el7.noarch

Smart-Proxy Server: CentOS Linux release 7.9.2009 (Core)
foreman-proxy-content-3.18.3-1.el7.noarch

Repo that is being synced from the Foreman Server to the Smart Proxy:
centos/7/os/x86_64/

Error that shows in logs:
(Same as described by Thulium-Drake)
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) ‘NoneType’ object has no attribute ‘findall’
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) Traceback (most recent call last):
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 294, in run
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) self.update_content(metadata_files, url)
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py”, line 627, in update_content
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) purge.purge_unwanted_units(metadata_files, self.conduit, self.config, catalog)
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py”, line 46, in purge_unwanted_units
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) remove_missing_groups(metadata_files, conduit)
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py”, line 165, in remove_missing_groups
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) remote_named_tuples = get_remote_units(file_function, group.GROUP_TAG, process_func)
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/purge.py”, line 277, in get_remote_units
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) for unit in package_info_generator:
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/packages.py”, line 64, in package_list_generator
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) package_info = process_func(element)
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py”, line 32, in process_group_element
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) packagelist.findall(‘packagereq’))
Jun 27 21:39:30 lab0l06 pulp: pulp_rpm.plugins.importers.yum.sync:ERROR: [df110a61] (1628-21760) AttributeError: ‘NoneType’ object has no attribute ‘findall’
Jun 27 21:39:30 lab0l06 pulp: pulp.server.async.tasks:INFO: [df110a61] Task failed : [df110a61-4abc-4f9b-8d34-bab7dc8f3666]
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) Task pulp.server.managers.repo.sync.sync[df110a61-4abc-4f9b-8d34-bab7dc8f3666] raised unexpected: PulpExecutionException(‘Importer indicated a failed response’,)
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) Traceback (most recent call last):
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 367, in trace_task
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) R = retval = fun(*args, **kwargs)
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 688, in call
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) return super(Task, self).call(*args, **kwargs)
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py”, line 110, in call
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) return super(PulpTask, self).call(*args, **kwargs)
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/celery/app/trace.py”, line 622, in protected_call
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) return self.run(*args, **kwargs)
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) File “/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py”, line 860, in sync
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) raise pulp_exceptions.PulpExecutionException(_(‘Importer indicated a failed response’))
Jun 27 21:39:30 lab0l06 pulp: celery.app.trace:ERROR: [df110a61] (1628-21760) PulpExecutionException: Importer indicated a failed response

Thulium-Drake’s workaround of modifying /usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/repomd/group.py does in fact work but isn’t something I could deploy to a production environment.

I notice this thread hasn’t seen any activity since December is there another thread or an open issue I can refer to for updates?

Thulium-Drake · June 28, 2021, 12:14pm

Thanks for your research! Did you by any chance get the opportunity to test this with katello 4? I haven’t had time to do that yet…

@cintrix84 were you able to look into this further?

Thanks!

Mamba · June 29, 2021, 3:28am

I was successful in my effort to sync a repo from the main foreman to a smart proxy on version 4.0.1-1.
I’ll give 4.1 a test soon.

Here are my notes:

Foreman Server:
CentOS Linux release 7.9.2009 (Core)
kernel-3.10.0-1160.el7.x86_64
katello-common-4.0.1-1.el7.noarch

SELinux: Permissive (Due to the following)
#============= pulpcore_server_t ==============
allow pulpcore_server_t httpd_config_t:dir search;
allow pulpcore_server_t kernel_t:unix_dgram_socket sendto;
allow pulpcore_server_t self:unix_dgram_socket { connect create };

Smart-Proxy Server:
CentOS Linux release 7.9.2009 (Core)
kernel-3.10.0-1160.el7.x86_64
foreman-proxy-content-4.0.1-1.el7.noarch

SELinux: Permissive (Due to the following)
#============= pulpcore_server_t ==============
allow pulpcore_server_t httpd_config_t:dir search;
allow pulpcore_server_t kernel_t:unix_dgram_socket sendto;
allow pulpcore_server_t self:unix_dgram_socket { connect create };
allow pulpcore_server_t var_lib_t:dir { add_name write };
allow pulpcore_server_t var_lib_t:file { create getattr ioctl open read write };

Repo that is being synced from the Foreman Server to the Smart Proxy:
centos/7/os/x86_64/

Goal 1: Sync from Internet Repo - SUCCESS
Goal 2: Sync from Main Foreman to Smart Proxy - SUCCESS after receiving a warning on my first attempt (See Below)
Goal 3: Build a CentOS 7 using the Smart Proxy - SUCCESS

Received this warning on my first attempt at syncing from the Main Foreman to the Smart Proxy

duplicate key value violates unique constraint “rpm_image_name_path_platforms_dist_188721ef_uniq”
DETAIL: Key (name, path, platforms, distribution_tree_id)=(boot.iso, images/boot.iso, x86_64, 40a2315e-edfb-4729-8f4a-6bc41e33d91d) already exists.

Mamba · June 29, 2021, 7:38am

I did not have as much success with 4.1.0-1 as I did with 4.0.1-1

I performed a fresh install and tried to sync a repository from the internet to the main foreman server.

The attempt would get about this far and then exit with a warning:

Total steps: 7907/12973

Associating Content: 2000/2000
Downloading Artifacts: 788/788
Downloading Metadata Files: 4/4
Parsed Comps: 109/109
Parsed Packages: 5006/10072
Un-Associating Content: 0/0

Note: I tried about 6 different mirrors (all geographically close to me) and all would fail with the same warning.

I never got to the point where I could test syncing from the main foreman server to a smart proxy.

Here are my notes:
Foreman Server:
CentOS Linux release 7.9.2009 (Core)
kernel 3.10.0-1160.31.1.el7.x86_64 #1 SMP
katello-4.1.0-1.el7.noarch
foreman-2.5.1-1.el7.noarch
4 Cores / 20GiB Memory

SELinux: Permissive (Due to the following)
#============= pulpcore_server_t ==============
allow pulpcore_server_t httpd_config_t:dir search;
allow pulpcore_server_t kernel_t:unix_dgram_socket sendto;
allow pulpcore_server_t self:unix_dgram_socket { connect create };

Repo that is being synced from the Internet:
centos/7/os/x86_64/

Issues with 4.x feel out of scope with this thread so I’ll just post 2 of the more interesting lines from the logs in case it helps someone not feel so lonely with their sync issue.

File “/usr/lib64/python3.6/site-packages/aiohttp/helpers.py”, line 656, in exit\n" +
" raise asyncio.TimeoutError from None\n",

and

“poll_attempts”=>{“total”=>40, “failed”=>1}}

Mamba · July 3, 2021, 1:50am

I did want to note that while I was successful syncing to and building a CentOS 7 host from a 4.0 Smart Proxy, I did encounter a bug where anaconda wouldn’t pull the packages from the smart proxy during the build until I ran a recursive wget against all of the files in the OS repository.

This wasn’t an issue building directly from the main foreman server.

Nor was this was an issue for the CentOS 8 Stream test build I did, it built no problem from the smart proxy.