Pulp_streamer many downloads fail from cdn.redhat.com

Support
#1

Problem:
Syncing red hat repositories often fails, full verified sync on a repository will show successful when many packages were not downloaded.

Files that are missing will be attempted when client makes a request for the package to install, but many failures will still occur, often ending in the client install operation failing or taking a considerable amount of time to complete.

Expected outcome:
Sync and verify will verify the downloads are complete. If cdn.redhat.com can’t provide the package, there should be some visibility of this, and the ability to sync a list of individual packages repeatedly until they are on disk.

Foreman and Proxy versions:
Katello 3.12

Foreman and Proxy plugin versions:
[root@foreman001 ~]# rpm -qa | grep pulp
python-pulp-client-lib-2.19.1-1.el7.noarch
pulp-docker-plugins-3.2.4-1.el7.noarch
python-pulp-puppet-common-2.19.1-1.el7.noarch
python-pulp-common-2.19.1-1.el7.noarch
pulp-server-2.19.1-1.el7.noarch
pulp-rpm-admin-extensions-2.19.1-1.el7.noarch
python-pulp-deb-common-1.9.1-1.el7.noarch
python-pulp-docker-common-3.2.4-1.el7.noarch
python-pulp-streamer-2.19.1-1.el7.noarch
pulp-rpm-handlers-2.19.1-1.el7.noarch
pulp-puppet-tools-2.19.1-1.el7.noarch
python-pulp-bindings-2.19.1-1.el7.noarch
python-pulp-repoauth-2.19.1-1.el7.noarch
pulp-deb-plugins-1.9.1-1.el7.noarch
python-pulp-oid_validation-2.19.1-1.el7.noarch
pulp-puppet-plugins-2.19.1-1.el7.noarch
rubygem-smart_proxy_pulp-1.4.1-1.el7.noarch
pulp-admin-client-2.19.1-1.el7.noarch
pulp-katello-1.0.2-1.el7.noarch
python-pulp-agent-lib-2.19.1-1.el7.noarch
pulp-rpm-plugins-2.19.1-1.el7.noarch
python-pulp-rpm-common-2.19.1-1.el7.noarch
pulp-selinux-2.19.1-1.el7.noarch

Other relevant data:
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)

Sep 16 13:01:05 foreman001 pulp_streamer: nectar.downloaders.threaded:WARNING: Request Timeout - Connection with https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-hook-vmfex-dev-4.30.24-2.el7ev.noarch.rpm timed out.
Sep 16 13:01:05 foreman001 pulp_streamer: pulp.streamer.server:INFO: Download failed []: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-hook-vmfex-dev-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:01:05 foreman001 pulp_streamer: pulp.streamer.server:ERROR: All download attempts failed: /var/lib/pulp/content/units/rpm/de/6099bfec4f9e36bce37b14258eb13a21c637bdbfb8b4658c6367927938fc29/vdsm-hook-vmfex-dev-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:01:05 foreman001 pulp_streamer: [-] 127.0.0.1 - - [16/Sep/2019:20:01:04 +0000] "GET /var/lib/pulp/content/units/rpm/de/6099bfec4f9e36bce37b14258eb13a21c637bdbfb8b4658c6367927938fc29/vdsm-hook-vmfex-dev-4.30.24-2.el7ev.noarch.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.10 yum/3.4.3"
Sep 16 13:01:05 foreman001 pulp_streamer: pulp.streamer.server:INFO: Trying URL: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-http-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:01:05 foreman001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (62): cdn.redhat.com
Sep 16 13:01:32 foreman001 pulp_streamer: nectar.downloaders.threaded:WARNING: Request Timeout - Connection with https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-http-4.30.24-2.el7ev.noarch.rpm timed out.
Sep 16 13:01:32 foreman001 pulp_streamer: pulp.streamer.server:INFO: Download failed []: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-http-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:01:32 foreman001 pulp_streamer: pulp.streamer.server:ERROR: All download attempts failed: /var/lib/pulp/content/units/rpm/eb/3689fb2d7ed05d325f5a360d235efbd8b2c26c048404cb8ade818a7433fa1e/vdsm-http-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:01:32 foreman001 pulp_streamer: [-] 127.0.0.1 - - [16/Sep/2019:20:01:31 +0000] "GET /var/lib/pulp/content/units/rpm/eb/3689fb2d7ed05d325f5a360d235efbd8b2c26c048404cb8ade818a7433fa1e/vdsm-http-4.30.24-2.el7ev.noarch.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.10 yum/3.4.3"
Sep 16 13:01:32 foreman001 pulp_streamer: pulp.streamer.server:INFO: Trying URL: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-jsonrpc-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:01:32 foreman001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (63): cdn.redhat.com
Sep 16 13:02:00 foreman001 pulp_streamer: nectar.downloaders.threaded:WARNING: Request Timeout - Connection with https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-jsonrpc-4.30.24-2.el7ev.noarch.rpm timed out.
Sep 16 13:02:00 foreman001 pulp_streamer: pulp.streamer.server:INFO: Download failed []: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-jsonrpc-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:02:00 foreman001 pulp_streamer: pulp.streamer.server:ERROR: All download attempts failed: /var/lib/pulp/content/units/rpm/5e/b4b91b331104d613172b59808aa97ca28f59cccb1d57a98333bef8c76cb192/vdsm-jsonrpc-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:02:00 foreman001 pulp_streamer: [-] 127.0.0.1 - - [16/Sep/2019:20:01:59 +0000] "GET /var/lib/pulp/content/units/rpm/5e/b4b91b331104d613172b59808aa97ca28f59cccb1d57a98333bef8c76cb192/vdsm-jsonrpc-4.30.24-2.el7ev.noarch.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.10 yum/3.4.3"
Sep 16 13:02:00 foreman001 pulp_streamer: pulp.streamer.server:INFO: Trying URL: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-network-4.30.24-2.el7ev.x86_64.rpm
Sep 16 13:02:00 foreman001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (64): cdn.redhat.com
Sep 16 13:02:27 foreman001 pulp_streamer: nectar.downloaders.threaded:WARNING: Request Timeout - Connection with https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-network-4.30.24-2.el7ev.x86_64.rpm timed out.
Sep 16 13:02:27 foreman001 pulp_streamer: pulp.streamer.server:INFO: Download failed []: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-network-4.30.24-2.el7ev.x86_64.rpm
Sep 16 13:02:27 foreman001 pulp_streamer: pulp.streamer.server:ERROR: All download attempts failed: /var/lib/pulp/content/units/rpm/7c/e7a3f9ab557aebed4af1f2d10d448bf873d726f200d9e2528496c39c8efd50/vdsm-network-4.30.24-2.el7ev.x86_64.rpm
Sep 16 13:02:27 foreman001 pulp_streamer: [-] 127.0.0.1 - - [16/Sep/2019:20:02:26 +0000] "GET /var/lib/pulp/content/units/rpm/7c/e7a3f9ab557aebed4af1f2d10d448bf873d726f200d9e2528496c39c8efd50/vdsm-network-4.30.24-2.el7ev.x86_64.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.10 yum/3.4.3"
Sep 16 13:02:27 foreman001 pulp_streamer: pulp.streamer.server:INFO: Trying URL: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-python-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:02:27 foreman001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (65): cdn.redhat.com
Sep 16 13:02:55 foreman001 pulp_streamer: nectar.downloaders.threaded:WARNING: Request Timeout - Connection with https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-python-4.30.24-2.el7ev.noarch.rpm timed out.
Sep 16 13:02:55 foreman001 pulp_streamer: pulp.streamer.server:INFO: Download failed []: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-python-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:02:55 foreman001 pulp_streamer: pulp.streamer.server:ERROR: All download attempts failed: /var/lib/pulp/content/units/rpm/aa/1794b6d28f0e2d930c227aa26d7eb957f7f94499150c1ef03a9ec2d6ebafcb/vdsm-python-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:02:55 foreman001 pulp_streamer: [-] 127.0.0.1 - - [16/Sep/2019:20:02:54 +0000] "GET /var/lib/pulp/content/units/rpm/aa/1794b6d28f0e2d930c227aa26d7eb957f7f94499150c1ef03a9ec2d6ebafcb/vdsm-python-4.30.24-2.el7ev.noarch.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.10 yum/3.4.3"
Sep 16 13:02:55 foreman001 pulp_streamer: pulp.streamer.server:INFO: Trying URL: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-yajsonrpc-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:02:55 foreman001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (66): cdn.redhat.com
Sep 16 13:03:22 foreman001 pulp_streamer: nectar.downloaders.threaded:WARNING: Request Timeout - Connection with https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-yajsonrpc-4.30.24-2.el7ev.noarch.rpm timed out.
Sep 16 13:03:22 foreman001 pulp_streamer: pulp.streamer.server:INFO: Download failed []: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vdsm-yajsonrpc-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:03:22 foreman001 pulp_streamer: pulp.streamer.server:ERROR: All download attempts failed: /var/lib/pulp/content/units/rpm/f4/f668914df963ec1c00f817e82a55230e7887e1eb63d4ddf5d7fef0e7660a49/vdsm-yajsonrpc-4.30.24-2.el7ev.noarch.rpm
Sep 16 13:03:22 foreman001 pulp_streamer: [-] 127.0.0.1 - - [16/Sep/2019:20:03:21 +0000] "GET /var/lib/pulp/content/units/rpm/f4/f668914df963ec1c00f817e82a55230e7887e1eb63d4ddf5d7fef0e7660a49/vdsm-yajsonrpc-4.30.24-2.el7ev.noarch.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.10 yum/3.4.3"
Sep 16 13:03:22 foreman001 pulp_streamer: pulp.streamer.server:INFO: Trying URL: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vhostmd-0.5-13.el7.x86_64.rpm
Sep 16 13:03:22 foreman001 pulp_streamer: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (67): cdn.redhat.com
Sep 16 13:03:49 foreman001 pulp_streamer: nectar.downloaders.threaded:WARNING: Request Timeout - Connection with https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vhostmd-0.5-13.el7.x86_64.rpm timed out.
Sep 16 13:03:49 foreman001 pulp_streamer: pulp.streamer.server:INFO: Download failed []: https://cdn.redhat.com/content/dist/rhel/server/7/7Server/x86_64/rhv-mgmt-agent/4/os/Packages/v/vhostmd-0.5-13.el7.x86_64.rpm
Sep 16 13:03:49 foreman001 pulp_streamer: pulp.streamer.server:ERROR: All download attempts failed: /var/lib/pulp/content/units/rpm/6e/5a48b1fa40c376210870740cd5661ee91079465c4de01ecd003d5c8ed1ee63/vhostmd-0.5-13.el7.x86_64.rpm
Sep 16 13:03:49 foreman001 pulp_streamer: [-] 127.0.0.1 - - [16/Sep/2019:20:03:49 +0000] "GET /var/lib/pulp/content/units/rpm/6e/5a48b1fa40c376210870740cd5661ee91079465c4de01ecd003d5c8ed1ee63/vhostmd-0.5-13.el7.x86_64.rpm HTTP/1.1" 404 - "-" "urlgrabber/3.10 yum/3.4.3"
#2

Red Hat CDN is currently having some issues last couple of days, please contact Red Hat support in order to find more.

#3

More than this week though? I opened a ticket end of last week and they said everything is OK.

Is there a way to sync a repo to local disk using commonly available tools so I can remove pulp as the culprit?

#4

https://status.redhat.com/ shows that there is still a problem. Based on other users i’ve heard from, I think this is likely the case.

You could probably try using ‘mrepo’, but you’d have to get the entitlement certs out of the manifest and sync the path directly. i’m not sure of any other tool.