Pulp deb repo sync error

Problem:
Testing Debian repository sync to pull down Ubuntu packages. Sync starts retrieving units and ends up failing part of the way through.

    {"num_success"=>1879,
     "description"=>"Retrieving units",
     "step_type"=>"sync_step_unit_download",
     "items_total"=>6858,
     "state"=>"FAILED",
     "error_details"=>[],
     "details"=>"",
     "num_failures"=>2,
     "step_id"=>"09ee775f-9071-4476-8854-d68d40aef324",
     "num_processed"=>1879},

Repo Sync Settings:
Upstream URL http://archive.ubuntu.com/ubuntu
Releases xenial
Components main
Architectures amd64
Verify SSL No

After 15 minutes, it fails with this error:
PLP0000: [Errno 2] No such file or directory: u’/var/cache/pulp/reserved_resource_worker-5@servername/2857703e-68a8-4a4c-a50e-c85916ae6290/packages/08/25/libreoffice-gtk3_5.1.2-0ubuntu1_amd64.deb’

Expected outcome:
Pulp should sucessfully sync
Foreman and Proxy versions:
Foreman 1.17 RC2 Katello 3.6 RC2

This looks like 2 Packages failed to download properly. The only way, i know of to remedy this situation is to delete the corresponding content_units in the mongodb, and restart the sync (assuming, the download is successful on second attempt).

If i recall correctly, you need to figure out the storage path of the failed units and fire up a mongodb-shell. There you can delete them by

> db.runCommand({ delete: "units_deb", deletes: [{q: {_storage_path: "<storage_path>"}, limit: 1}]})

Of course, if you know of another way to identify the failed units, you can adjust the search pattern after q:.
Good luck there!

Thanks for the tip. I’ll try and see if I can figure this out.
Also, when watching journalctl -f during a sync, pulp gives these errors when it fails:

Apr 12 11:05:01 servername pulp[45017]: py.warnings:WARNING: [1de2dcce] (45017-69216) /usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py:1277: DeprecationWarning: save is deprecated. Use insert_one or replace_one instead
Apr 12 11:05:01 servername pulp[45017]: py.warnings:WARNING: [1de2dcce] (45017-69216) publish_result_coll.save(result)
Apr 12 11:05:01 servername pulp[45017]: py.warnings:WARNING: [1de2dcce] (45017-69216)
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216) Exception caught from plugin during publish for repo [3ad83a9d-518a-43ed-be91-799cd1b5db82]
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216) Traceback (most recent call last):
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1258, in_do_publish
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)     publish_report = publish_repo(transfer_repo, conduit, call_config)
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 737, in wrap_f
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)     return f(*args, **kwargs)
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 80, in publish_repo
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)     return self._publisher.process_lifecycle()
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 572, in process_lifecycle
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)     super(PluginStep, self).process_lifecycle()
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 163, in process_lifecycle
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)     step.process()
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 256, in process
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)     self._process_block()
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 303, in _process_block
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)     self.process_main()
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)   File "/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py", line 329, in process_main
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)     with_symlinks=True)
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)   File "/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py", line 477, in create
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)     component=component, architecture=architecture)
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)   File "/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py", line 447, in add_packages
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216)     sz = str(os.stat(filename).st_size)
Apr 12 11:05:01 servername pulp[45017]: pulp.server.controllers.repository:ERROR: [1de2dcce] (45017-69216) OSError: [Errno 2] No such file or directory: '/var/lib/pulp/content/units/deb/7e/24cd262538c6363288d6cbd53d6db55ee04c0991cbdf6405a5ce716f27da14/libjana-ecal0_0.0.0+git20091215.9ec1da8a-4+build3_amd64.deb'

Well there you have it. Part of this (i do not remember which part though) is the storage_path you need for the query.

Hello,

I’m still having issues pulling down Ubuntu repos. Just pulling down main,restricted,multiverse components work fine, but when pulling down universe component, it breaks every time.

When pulling all 4 components from a local repo, I get this error:

       {"num_success"=>0,
        "description"=>"",
        "step_type"=>"publish_repodata",
        "items_total"=>1,
        "state"=>"FAILED",
        "error_details"=>
         [{"traceback"=>
            "  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 256, in process\n" +
            "    self._process_block()\n" +
            "\n" +
            "  File \"/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py\", line 303, in _process_block\n" +
            "    self.process_main()\n" +
            "\n" +
            "  File \"/usr/lib/python2.7/site-packages/pulp_deb/plugins/distributors/distributor.py\", line 329, in process_main\n" +
            "    with_symlinks=True)\n" +
            "\n" +
            "  File \"/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py\", line 477, in create\n" +
            "    component=component, architecture=architecture)\n" +
            "\n" +
            "  File \"/usr/lib/python2.7/site-packages/debpkgr/aptrepo.py\", line 448, in add_packages\n" +
            "    pkg = debpkg.DebPkg.from_file(filename, Size=sz)\n" +
            "\n" +
            "  File \"/usr/lib/python2.7/site-packages/debpkgr/debpkg.py\", line 357, in from_file\n" +
            "    md5sums = debpkg.md5sums(encoding='utf-8')\n" +
            "\n" +
            "  File \"/usr/lib/python2.7/site-packages/debian/debfile.py\", line 323, in md5sums\n" +
            "    return self.control.md5sums(encoding=encoding, errors=errors)\n" +
            "\n" +
            "  File \"/usr/lib/python2.7/site-packages/debian/debfile.py\", line 245, in md5sums\n" +
            "    for line in md5_file.readlines():\n" +
            "\n" +
            "  File \"/usr/lib64/python2.7/codecs.py\", line 796, in readlines\n" +
            "    data = self.reader.read()\n" +
            "\n" +
            "  File \"/usr/lib64/python2.7/codecs.py\", line 477, in read\n" +
            "    newchars, decodedbytes = self.decode(data, self.errors)\n",
           "error"=>
            "'utf8' codec can't decode byte 0xed in position 107: invalid continuation byte"}],
        "details"=>"",
        "num_failures"=>1,
        "step_id"=>"ef321e38-f3fd-4658-b940-c56d146d612f",
        "num_processed"=>1}],
     "items_total"=>1,
     "state"=>"FAILED",
     "error_details"=>[{"traceback"=>nil, "error"=>nil}],
     "details"=>"",
     "num_failures"=>1,
     "step_id"=>"bf3e941f-329e-4d9c-99f2-49492e4127a2",
     "num_processed"=>1},

When pulling the 4 components from http://archive.ubuntu.com/ubuntu, I get this checksum error:

Katello::Errors::PulpError: DEBSYNC002: Unable to sync 1a54ffbf-d1a4-49f1-a78b-c841f70f465d from http://archive.ubuntu.com/ubuntu: mismatching checksums for roundcube-core_1.2~beta+dfsg.1-0ubuntu1_all.deb: expected 2c07729047ebc9206038d90c88d77e392f68fcfc5a3b7cb30cfd6e7cf15157c3, actual 5da63fcdcaab24853af20696edb6f44240ace3cc7e556639b4d082bbc44eb07b

See https://github.com/sassoftware/python-debpkgr/pull/8