CentOS 7 repo sync stops

Problem:

syncing of reps wait at 5% for few min then stop with “warning”
error output is bellow.

Expected outcome:
some packages being put in repo.

Foreman and Proxy versions:
katello 3.8

Foreman and Proxy plugin versions:

Other relevant data: 100% Complete

Output:

No new packages.

Errors:

RPM1004: Error retrieving metadata: A connection error occurred

Actions::Pulp::Repository::Sync
Input:
{"pulp_id"=>"6feca025-aa07-4393-ba4a-2e810c3fa142",
 "task_id"=>nil,
 "source_url"=>nil,
 "options"=>{},
 "remote_user"=>"admin",
 "remote_cp_user"=>"admin"}
Output:
{"pulp_tasks"=>
  [{"exception"=>nil,
    "task_type"=>"pulp.server.managers.repo.sync.sync",
    "_href"=>"/pulp/api/v2/tasks/f9035e0f-f6ee-4db8-9fa0-c451d041ff55/",
    "task_id"=>"f9035e0f-f6ee-4db8-9fa0-c451d041ff55",
    "tags"=>
     ["pulp:repository:6feca025-aa07-4393-ba4a-2e810c3fa142",
      "pulp:action:sync"],
    "finish_time"=>"2018-11-29T20:22:57Z",
    "_ns"=>"task_status",
    "start_time"=>"2018-11-29T20:18:11Z",
    "traceback"=>
     "Traceback (most recent call last):\n" +
     "  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 367, in trace_task\n" +
     "    R = retval = fun(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 529, in __call__\n" +
     "    return super(Task, self).__call__(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 107, in __call__\n" +
     "    return super(PulpTask, self).__call__(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/celery/app/trace.py\", line 622, in __protected_call__\n" +
     "    return self.run(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py\", line 769, in sync\n" +
     "    sync_report = sync_repo(transfer_repo, conduit, call_config)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py\", line 737, in wrap_f\n" +
     "    return f(*args, **kwargs)\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/importer.py\", line 82, in sync_repo\n" +
     "    report = self._current_sync.run()\n" +
     "  File \"/usr/lib/python2.7/site-packages/pulp_rpm/plugins/importers/yum/sync.py\", line 335, in run\n" +
     "    reason=self.repomd_not_found_reason)\n" +
     "PulpCodedException: Error retrieving metadata: A connection error occurred\n",
    "spawned_tasks"=>[],
    "progress_report"=>
     {"yum_importer"=>
       {"content"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "details"=>
           {"rpm_total"=>0, "rpm_done"=>0, "drpm_total"=>0, "drpm_done"=>0},
          "size_total"=>0,
          "size_left"=>0,
          "items_left"=>0},
        "comps"=>{"state"=>"FINISHED"},
        "purge_duplicates"=>{"state"=>"FINISHED"},
        "distribution"=>
         {"items_total"=>0,
          "state"=>"FINISHED",
          "error_details"=>[],
          "items_left"=>0},
        "errata"=>{"state"=>"FINISHED"},
        "metadata"=>{"state"=>"FINISHED"}}},
    "queue"=>
     "reserved_resource_worker-2@ip-10-112-242-57.nonprod.intl-xxx.xxx.dq2",
    "state"=>"error",
    "worker_name"=>
     "reserved_resource_worker-2@ip-10-112-242-57.nonprod.intl-xxx.xxx",
    "result"=>nil,
    "error"=>
     {"code"=>"RPM1004",
      "data"=>{"reason"=>"A connection error occurred"},
      "description"=>"Error retrieving metadata: A connection error occurred",
      "sub_errors"=>[]},
    "_id"=>{"$oid"=>"5c00498398b425286fdd88f0"},
    "id"=>"5c00498398b425286fdd88f0"}],
 "contents_changed"=>true,
 "poll_attempts"=>{"total"=>38, "failed"=>1}}
Action:
Actions::Katello::Repository::IndexContent
Input:
{"id"=>6,
 "contents_changed"=>Step(5).output[:contents_changed],
 "full_index"=>false,
 "current_user_id"=>4}
Output:
{}
Action:
Actions::Katello::Repository::ErrataMail
Input:
{"repo"=>6,
 "last_updated"=>"2018-11-29 20:18:11 +0000",
 "contents_changed"=>Step(5).output[:contents_changed]}
Output:
{}
1 Like

I’m seeing this too. Previously I’ve got around it by running the hammer script by hand, but that is now failing too. I’ve tried against these mirrors to check:

I looked at the /var/log/messages when katello was trying to sycn.
I saw following errors.

Nov 30 16:37:23 10 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (3): dl.fedoraproject.org
Nov 30 16:37:23 10 pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=3, connect=5, read=5, redirect=None)) after connection broken by ‘ProxyError(‘Cannot connect to proxy.’, gaierror(-2, ‘Name or service not known’))’: http://mirror.centos.org/centos/7/os/x86_64/repodata/repomd.xml
Nov 30 16:37:23 10 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (3): proxy=https
Nov 30 16:38:07 10 pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=2, connect=5, read=5, redirect=None)) after connection broken by ‘ProxyError(‘Cannot connect to proxy.’, gaierror(-2, ‘Name or service not known’))’: http://mirror.centos.org/centos/7/os/x86_64/repodata/repomd.xml
Nov 30 16:38:07 10 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (4): proxy=https
Nov 30 16:38:07 10 pulp: requests.packages.urllib3.connectionpool:WARNING: Retrying (Retry(total=2, connect=5, read=5, redirect=None)) after connection broken by ‘ProxyError(‘Cannot connect to proxy.’, gaierror(-2, ‘Name or service not known’))’: /pub/epel/6/x86_64/repodata/repomd.xml
Nov 30 16:38:07 10 pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (4): dl.fedoraproject.org

I have put proxy setting in

/etc/rc.local
/etc/profile.d/csh.local
/etc/profile.d/sh.local
and when I log in I can do

Blockquote
[root@10 ~]# curl http://mirror.centos.org/centos/7/os/x86_64/repodata/repomd.xml

<?xml version="1.0" encoding="UTF-8"?> 1525378614 d87379a47bc2060f833000b9cef7f9670195fe197271d37fce5791e669265e8b 1525378657 912297 3b7ec5d1dd09f452a54bacf567f486771f3333400169baa8f62d8107d3d6f2aa fce55daefb3fb1f3d03ac19e2f7e14ec5e479412efdd3f4d6300de8f6c370410 1525378639 7133920 100150821 29b154c359eaf12b9e35d0d5c649ebd62ce43333f39f02f33ed7b08c3b927e20 d87379a47bc2060f833000b9cef7f9670195fe197271d37fce5791e669265e8b 1525378657 169476 6afcd31233c69da026f363eece2a99e5a79842313fef23bc76b96b69e90dd49d c7f812b461e786c4d4f01b1ff4fd1f4767149c621970dc23f42826b72fff8aa1 1525378639 2916112 27233080 03d0a660eb33174331aee3e077e11d4c017412d761b7f2eaa8555e7898e701e0 5df1fecbf7db56d13110af1989c3db5ec4b31fcfab610108dc1efe78ff55111b 1525378657 10 6161792 30611456 ede157be4cf5c030483ab639f8f18c6daf268d4c233037a1578b3ed8258fa461 19bc02808e99acbe8ea531a4078a68a65409a422dab8b08602a5bf7a27e3ddce 1525378642 10 2669706 19270656 430581635daba01e8b69ff4227305a67f7f7d967d2313277f7368aab8c338d36 7b2d11bc1fcec6786d1185911f6cedc4ba04e491b85fcd3575e3e814bce22034 1525378639 1617685 20561270 f35d0029ad59e9fca0823be708150f5ee28ee1707cdc9bdaf420da676daafe28 b94684525280f6810de20424ca5affe7776e9ca90bf8ec3704dac511d4fbc34d 1525378651 10 7202122 46503936 [root@10 ~]#

Seems like katello is not being able to use proxy settings.

Will there be a response from the developers on this one soon? It’s an ongoing problem, for me it’s not sync, it’s just 404:

[EDIT] OH. I see the error now.

This is from /var/log/messages when I run

hammer repository synchronize --id 5 --source-url http://fedora.melbourneitmirror.net/epel/7/x86_64/

Dec  3 08:39:35 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:39:35 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/repodata/repomd.xml failed with code 404: Not Found
Dec  3 08:39:36 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: [979e4ed9] Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:39:36 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: [979e4ed9] Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/ failed with code 404: Not Found
Dec  3 08:39:36 vmpr-res-utils pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [979e4ed9] Downloading metadata from http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/.
Dec  3 08:39:36 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:39:36 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/repodata/repomd.xml failed with code 404: Not Found
Dec  3 08:39:37 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: [979e4ed9] Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:39:37 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: [979e4ed9] Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/ failed with code 404: Not Found
Dec  3 08:39:37 vmpr-res-utils pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [979e4ed9] Downloading metadata from http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/.
Dec  3 08:39:37 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:39:37 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/repodata/repomd.xml failed with code 404: Not Found
Dec  3 08:39:38 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: [979e4ed9] Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:39:38 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: [979e4ed9] Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/ failed with code 404: Not Found
Dec  3 08:39:38 vmpr-res-utils pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [979e4ed9] Downloading metadata from http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/.
Dec  3 08:39:38 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:39:38 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/repodata/repomd.xml failed with code 404: Not Found
Dec  3 08:39:39 vmpr-res-utils pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [979e4ed9] Downloading additional units.
Dec  3 08:39:39 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:39:39 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/.treeinfo failed with code 404: Not Found
Dec  3 08:39:40 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:39:40 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/treeinfo failed with code 404: Not Found
Dec  3 08:39:41 vmpr-res-utils pulp: pulp.server.async.tasks:INFO: [979e4ed9] Task failed : [979e4ed9-5b2c-4541-a52d-3245cbfd3998] : Error retrieving metadata: Not Found
Dec  3 08:39:41 vmpr-res-utils pulp: celery.app.trace:INFO: [979e4ed9] Task pulp.server.managers.repo.sync.sync[979e4ed9-5b2c-4541-a52d-3245cbfd3998] raised expected: PulpCodedException()
Dec  3 08:39:41 vmpr-res-utils pulp: celery.app.trace:INFO: [cff787fa] Task pulp.server.async.tasks._release_resource[cff787fa-7be7-4fdf-af1e-6abf89dfd4c3] succeeded in 0.00283853300016s: None

and this is from /var/log/messages when I run
hammer repository synchronize --id 5 --source-url http://epel.mirror.digitalpacific.com.au/7/x86_64/

Dec  3 08:42:54 vmpr-res-utils pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [4aa138fa] Downloading metadata from http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/.
Dec  3 08:42:54 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:42:54 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/repodata/repomd.xml failed with code 404: Not Found
Dec  3 08:42:55 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: [4aa138fa] Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:42:55 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: [4aa138fa] Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/ failed with code 404: Not Found
Dec  3 08:42:55 vmpr-res-utils pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [4aa138fa] Downloading metadata from http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/.
Dec  3 08:42:55 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:42:55 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/repodata/repomd.xml failed with code 404: Not Found
Dec  3 08:42:56 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: [4aa138fa] Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:42:56 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: [4aa138fa] Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/ failed with code 404: Not Found
Dec  3 08:42:56 vmpr-res-utils pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [4aa138fa] Downloading metadata from http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/.
Dec  3 08:42:56 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:42:56 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/repodata/repomd.xml failed with code 404: Not Found
Dec  3 08:42:57 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: [4aa138fa] Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:42:57 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: [4aa138fa] Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/ failed with code 404: Not Found
Dec  3 08:42:57 vmpr-res-utils pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [4aa138fa] Downloading metadata from http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/.
Dec  3 08:42:57 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:42:57 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/repodata/repomd.xml failed with code 404: Not Found
Dec  3 08:42:58 vmpr-res-utils pulp: pulp_rpm.plugins.importers.yum.sync:INFO: [4aa138fa] Downloading additional units.
Dec  3 08:42:58 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:42:58 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/.treeinfo failed with code 404: Not Found
Dec  3 08:42:59 vmpr-res-utils pulp: requests.packages.urllib3.connectionpool:INFO: Starting new HTTP connection (1): fedora.uberglobalmirror.com
Dec  3 08:42:59 vmpr-res-utils pulp: nectar.downloaders.threaded:INFO: Download failed: Download of http://fedora.uberglobalmirror.com/pub/epel/7/x86_64/treeinfo failed with code 404: Not Found
Dec  3 08:43:00 vmpr-res-utils pulp: pulp.server.async.tasks:INFO: [4aa138fa] Task failed : [4aa138fa-3098-4d69-a2b1-f541fd936cec] : Error retrieving metadata: Not Found
Dec  3 08:43:00 vmpr-res-utils pulp: celery.app.trace:INFO: [4aa138fa] Task pulp.server.managers.repo.sync.sync[4aa138fa-3098-4d69-a2b1-f541fd936cec] raised expected: PulpCodedException()
Dec  3 08:43:00 vmpr-res-utils pulp: celery.app.trace:INFO: [387e489e] Task pulp.server.async.tasks._release_resource[387e489e-32d2-4ab2-aebb-abd8b56793d9] succeeded in 0.00251583099998s: None

So it looks like hammer is ignoring --source-url

hammer --version
hammer (0.15.1)

  • hammer_cli_foreman (0.15.1)
  • hammer_cli_foreman_bootdisk (0.1.3)
  • hammer_cli_foreman_docker (unknown version)
  • hammer_cli_foreman_tasks (unknown version)
  • hammer_cli_import (0.11.3)
  • hammer_cli_katello (0.15.0)

I feel like there is a snowball effect happening because the original bug hasn’t been addressed - this work around for that bug no longer works.

Bug report opened.