CentOS / Ubuntu Repositories fail to sync

Support
1

Problem:
Sync incomplete. I try multiple times to sync my repository with no success. I get some content but not all. Sync normally stops at 60% then go to incomplete.

Expected outcome:
complete

Foreman and Proxy versions:
Version 1.24.3

Foreman and Proxy plugin versions:
foreman-proxy-1.0-1.noarch
foreman-proxy-1.24.3-1.el7.noarch

Distribution and version:
CentOS Linux release 7.8.2003 (Core)

Other relevant data:
From the backend the foreman logs don’t show any obvious error only from the GUI > tasks > errors

{"pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.sync.sync",
    "_href"=>"/pulp/api/v2/tasks/84ed085f-0ba3-493c-be1f-eb3e742d15d8/",
    "task_id"=>"84ed085f-0ba3-493c-be1f-eb3e742d15d8",
    "tags"=>
     ["pulp:repository:757b29cb-8e78-487a-bb84-682b85a5ca33",
      "pulp:action:sync"],
    "finish_time"=>"2020-07-08T16:03:06Z",
    "_ns"=>"task_status",
    "start_time"=>"2020-07-08T15:57:09Z",
    "traceback"=>
     "Traceback (most recent call last):\n" +
     "  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" +
     "    R = retval = fun(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 686, in __call__\n" +
     "    return super(Task, self).__call__(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 108, in __call__\n" +
     "    return super(PulpTask, self).__call__(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" +
     "    return self.run(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 793, in sync\n" +
     "    sync_report = sync_repo(transfer_repo, conduit, call_config)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 894, in wrap_f\n" +
     "    return f(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/importer.py\", line 135, in sync_repo\n" +
     "    report = self._current_sync.process_lifecycle()\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 573, in process_lifecycle\n" +
     "    super(PluginStep, self).process_lifecycle()\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 164, in process_lifecycle\n" +
     "    step.process()\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 257, in process\n" +
     "    self._process_block()\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 304, in _process_block\n" +
     "    self.process_main()\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py\", line 360, in process_main\n" +
     "    checksums = unit.calculate_deb_checksums(path)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp_deb/plugins/db/models.py\", line 254, in calculate_deb_checksums\n" +
     "    with open(input_file_path) as input_file:\n" +
     "IOError: [Errno 2] No such file or directory: u'/var/cache/pulp/reserved_resource_worker-1@hq-katello/84ed085f-0ba3-493c-be1f-eb3e742d15d8/packages/75/61/python-django-app-plugins_0.1.1-2_all.deb'\n",
    "spawned_tasks"=>[],
    "progress_report"=>
     {"deb_importer"=>
       [{"num_success"=>2,
         "description"=>"Retrieving metadata: Release file(s)",
         "step_type"=>"sync_step_release_download",
         "items_total"=>2,
         "state"=>"FINISHED",
         "error_details"=>[],
         "details"=>"",
         "num_failures"=>0,
         "step_id"=>"47979566-9686-44e4-a707-6988ea77693d",
         "num_processed"=>2},
        {"num_success"=>1,
         "description"=>"Parse Release Files",
         "step_type"=>"sync_step_release_parse",
         "items_total"=>1,
         "state"=>"FINISHED",
         "error_details"=>[],
         "details"=>"",
         "num_failures"=>0,
         "step_id"=>"055360bd-282d-4078-a597-f3d3abaedc23",
         "num_processed"=>1},
        {"num_success"=>4,
         "description"=>"Retrieving metadata: Packages files",
         "step_type"=>"sync_step_packages_download",
         "items_total"=>4,
         "state"=>"FINISHED",
         "error_details"=>[],
         "details"=>"",
         "num_failures"=>0,
         "step_id"=>"6d2f3376-c7a6-4c3c-b1b7-33826c2a7791",
         "num_processed"=>4},
        {"num_success"=>1,
         "description"=>"Parse Packages Files",
         "step_type"=>"sync_step_packages_parse",
         "items_total"=>1,
         "state"=>"FINISHED",
         "error_details"=>[],
         "details"=>"",
         "num_failures"=>0,
         "step_id"=>"3c8a6832-1526-49d3-b1c0-86d3e0533ee7",
         "num_processed"=>1},
        {"num_success"=>1,
         "description"=>"Copying units already in pulp",
         "step_type"=>"get_local",
         "items_total"=>1,
         "state"=>"FINISHED",
         "error_details"=>[],
         "details"=>"",
         "num_failures"=>0,
         "step_id"=>"adafe190-2b33-42f5-9f01-7b8961a56ad3",
         "num_processed"=>1},
        {"num_success"=>1,
         "description"=>"Prepare Package Download",
         "step_type"=>"sync_step_unit_download_requests",
         "items_total"=>1,
         "state"=>"FINISHED",
         "error_details"=>[],
         "details"=>"",
         "num_failures"=>0,
         "step_id"=>"dd785808-d790-4dc4-bca1-6a2d350364e1",
         "num_processed"=>1},
        {"num_success"=>401,
         "description"=>"Retrieving units",
         "step_type"=>"sync_step_unit_download",
         "items_total"=>53837,
         "state"=>"FAILED",
         "error_details"=>[],
         "details"=>"",
         "num_failures"=>53436,
         "step_id"=>"05a3ab51-74e6-46a9-afbd-205c62ee6d24",
         "num_processed"=>53837},
        {"num_success"=>0,
         "description"=>"Save and associate downloaded units",
         "step_type"=>"sync_step_save",
         "items_total"=>1,
         "state"=>"FAILED",
         "error_details"=>
          [{"traceback"=>
             "  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 257, in process\n" +
             "    self._process_block()\n" +
             "\n" +
             "  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 304, in _process_block\n" +
             "    self.process_main()\n" +
             "\n" +
             "  File \"/usr/lib/python2.7/site-packages/pulp_deb/plugins/importers/sync.py\", line 360, in process_main\n" +
             "    checksums = unit.calculate_deb_checksums(path)\n" +
             "\n" +
             "  File \"/usr/lib/python2.7/site-packages/pulp_deb/plugins/db/models.py\", line 254, in calculate_deb_checksums\n" +
             "    with open(input_file_path) as input_file:\n",
            "error"=>
             "[Errno 2] No such file or directory: u'/var/cache/pulp/reserved_resource_worker-1@hq-katello/84ed085f-0ba3-493c-be1f-eb3e742d15d8/packages/75/61/python-django-app-plugins_0.1.1-2_all.deb'"}],
         "details"=>"",
         "num_failures"=>1,
         "step_id"=>"d8f2c2c0-99d9-4ed6-acf7-333562a37496",
         "num_processed"=>1},
        {"num_success"=>0,
         "description"=>"Save metadata",
         "step_type"=>"sync_step_save_meta",
         "items_total"=>1,
         "state"=>"NOT_STARTED",
         "error_details"=>[],
         "details"=>"",
         "num_failures"=>0,
         "step_id"=>"4ec14e7c-cc37-43d5-9630-84c976deda6d",
         "num_processed"=>0},
        {"num_success"=>0,
         "description"=>"Orphan removed units",
         "step_type"=>"sync_step_orphan_removed_units",
         "items_total"=>1,
         "state"=>"NOT_STARTED",
         "error_details"=>[],
         "details"=>"",
         "num_failures"=>0,
         "step_id"=>"f4704c90-0641-4dda-9ff2-e7895f61b2d4",
         "num_processed"=>0}]},
    "queue"=>"reserved_resource_worker-1@hq-katello.dq2",
    "state"=>"error",
    "worker_name"=>"reserved_resource_worker-1@hq-katello",
    "result"=>nil,
    "error"=>
     {"code"=>"PLP0000",
      "data"=>{},
      "description"=>
       "[Errno 2] No such file or directory: u'/var/cache/pulp/reserved_resource_worker-1@hq-katello/84ed085f-0ba3-493c-be1f-eb3e742d15d8/packages/75/61/python-django-app-plugins_0.1.1-2_all.deb'",
      "sub_errors"=>[]},
    "_id"=>{"$oid"=>"5f05ecd597013fbcc65f8747"},
    "id"=>"5f05ecd597013fbcc65f8747"}],
 "contents_changed"=>true,
 "poll_attempts"=>{"total"=>43, "failed"=>1}}


Actions::Katello::Repository::Sync

Input:

{"repository"=>{"id"=>6, "name"=>"Ubuntu_16-Repo", "label"=>"Ubuntu_16-Repo"},
 "product"=>
  {"id"=>3,
   "name"=>"Ubuntu_16",
   "label"=>"Ubuntu_16",
   "cp_id"=>"690946343391"},
 "provider"=>{"id"=>3, "name"=>"Anonymous"},
 "organization"=>{"id"=>3, "name"=>"pfcta", "label"=>"pfcta"},
 "services_checked"=>["pulp", "pulp_auth"],
 "id"=>6,
 "sync_result"=>Step(6).output,
 "skip_metadata_check"=>false,
 "validate_contents"=>false,
 "contents_changed"=>Step(6).output[:contents_changed],
 "current_request_id"=>nil,
 "current_timezone"=>"America/New_York",
 "current_user_id"=>4,
 "current_organization_id"=>3,
 "current_location_id"=>2}
2

Hey @pfcta01, really sorry about the delay!

Looks like a pulp issue. Just to rule out one cause, is the disk full?

3

Hey John,

Thanks for your reply on this. No, the disk is not full, this is a fresh install and has about 150G of storage available for /var/lib/pulp to utilize.

Off topic
Do you know of any known way to add Ubuntu servers to katello? I currently have my centOS servers registered but finding it very difficult to bootstrap or register ubuntu servers. I found that I may have to rebuild subscription manager to accomplish this. Can you point me in the right direction for this?

Thanks

4

@Justin_Sherrill any ideas on the original issue here? Looks like there could be an issue syncing deb repos

I know there has been some discussion around this before, but I’m not sure what is supported right now
@evgeni do you know?

5

I see this pulp issue, which seems to have a similar traceback: https://pulp.plan.io/issues/6240

6

No, I do not, maybe @x9c4 or @Bernhard_Suttner have pointers.

But my you don’t need subman on Debian. You can just use the unprotected repository URLs with regular apt.

7

does this file / path exist?

8

Yes but not the full path, it ends at:
/var/cache/pulp/reserved_resource_worker-1@hq-katello

Directory Empty

9

Hi, any possible solution to this? I still have not fixed the issue i’m experiencing since our last conversation.

Thanks

10

Can you provide all the details about the repository so we could try to reproduce? From the repo details page: Url, arch’s, components, releases

11

Also, you mentioned that the centos repos fail to sync, is it with a similar error?

Can you paste that error?

How much disk space do you have free on /var/lib/pulp and /var/cache/pulp ?

12

Hi,

Disregard my last request, katello mysteriously fixed itself and completed the sync.

I do have one issue though. I am having a hard time registering my Ubuntu server to katello. Since subscription-manager is null and void, I tried bootstrapping but nothing works. All my centOS servers are registered but not ubuntu. Can you steer me in the right direction on how to accomplish this task?

Thanks