Katello::Errors::Pulp3Error: Package matching query does not exist

**Problem:** 
I have a few OEL repos that fail to sync and give the error:

Katello::Errors::Pulp3Error: Package matching query does not exist.

repos failing to sync are from:

http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/
http://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/

They are downloaded via a http proxy which is being used by other repos that I have successfully syncing.

I have tried creating a new test product and repo and syncing but still have no luck.

Is there a way to remove the repo completely or force it to rebuild completely.
I have tried advanced sync all of the options but still no luck.

Is there some black magic I can do on the backend to try to see which package is causing the issue and or how to remove said package from the local copy. ?

I'm at a loss and need some help here. 


**Expected outcome:**
Repos sync as expected.  other OEL repos sync fine.

**Foreman and Proxy versions:**
foreman-2.1.4-1.el7.noarch
foreman-proxy-2.1.4-1.el7.noarch
katello-3.16.1-1.el7.noarch
pulp-server-2.21.4-2.el7.noarch

**Distribution and version:**
CentOS Linux release 7.9.2009 (Core)

# rpm -qa | grep -i pulp | sort
pulp-admin-client-2.21.4-2.el7.noarch
pulp-client-1.0-1.noarch
pulpcore-selinux-1.0.0-2.el7.x86_64
pulp-deb-plugins-1.10.2-1.el7.noarch
pulp-docker-plugins-3.2.8-1.el7.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-puppet-plugins-2.21.4-1.el7.noarch
pulp-puppet-tools-2.21.4-1.el7.noarch
pulp-rpm-admin-extensions-2.21.4-1.el7.noarch
pulp-rpm-plugins-2.21.4-1.el7.noarch
pulp-selinux-2.21.4-2.el7.noarch
pulp-server-2.21.4-2.el7.noarch
python3-pulp-2to3-migration-0.2.0-0.1.b7.el7.noarch
python3-pulp-certguard-0.1.0-0.2.rc5.el7.noarch
python3-pulp-container-1.4.2-1.el7.noarch
python3-pulpcore-3.4.1-1.el7.noarch
python3-pulp-file-1.0.1-1.el7.noarch
python3-pulp-rpm-3.5.1-1.el7.noarch
python-pulp-bindings-2.21.4-2.el7.noarch
python-pulp-client-lib-2.21.4-2.el7.noarch
python-pulp-common-2.21.4-2.el7.noarch
python-pulp-deb-common-1.10.2-1.el7.noarch
python-pulp-docker-common-3.2.8-1.el7.noarch
python-pulp-oid_validation-2.21.4-2.el7.noarch
python-pulp-puppet-common-2.21.4-1.el7.noarch
python-pulp-repoauth-2.21.4-2.el7.noarch
python-pulp-rpm-common-2.21.4-1.el7.noarch
python-pulp-streamer-2.21.4-2.el7.noarch
tfm-rubygem-pulp_2to3_migration_client-0.2.0-0.1.b6.el7.noarch
tfm-rubygem-pulp_ansible_client-0.2.0b13.dev01588546902-1.el7.noarch
tfm-rubygem-pulp_certguard_client-0.1.0rc5-1.el7.noarch
tfm-rubygem-pulp_container_client-1.4.1-1.el7.noarch
tfm-rubygem-pulpcore_client-3.4.1-1.el7.noarch
tfm-rubygem-pulp_file_client-1.0.1-1.el7.noarch
tfm-rubygem-pulp_rpm_client-3.5.0-1.el7.noarch
tfm-rubygem-smart_proxy_pulp-2.1.0-3.fm2_1.el7.noarch

# rpm -qa | grep -i katello | sort
foreman-installer-katello-2.1.4-1.el7.noarch
katello-3.16.1-1.el7.noarch
katello-agent-3.5.4-1.el7.noarch
katello-certs-tools-2.7.0-1.el7.noarch
katello-client-bootstrap-1.7.5-1.el7.noarch
katello-common-3.16.1-1.el7.noarch
katello-debug-3.16.1-1.el7.noarch
katello-default-ca-1.0-1.noarch
katello-host-tools-3.5.4-1.el7.noarch
katello-host-tools-fact-plugin-3.5.4-1.el7.noarch
katello-repos-3.16.1-1.el7.noarch
katello-selinux-3.4.0-1.el7.noarch
katello-server-ca-1.0-1.noarch
pulp-katello-1.0.3-1.el7.noarch
tfm-rubygem-hammer_cli_katello-0.22.2-1.el7.noarch
tfm-rubygem-katello-3.16.1.2-1.el7.noarch

# rpm -qa | grep -i foreman | sort
foreman-2.1.4-1.el7.noarch
foreman-cli-2.1.4-1.el7.noarch
foreman-debug-2.1.4-1.el7.noarch
foreman-dynflow-sidekiq-2.1.4-1.el7.noarch
foreman-installer-2.1.4-1.el7.noarch
foreman-installer-katello-2.1.4-1.el7.noarch
foreman-postgresql-2.1.4-1.el7.noarch
foreman-proxy-2.1.4-1.el7.noarch
foreman-proxy-content-3.16.1-1.el7.noarch
foreman-release-2.1.4-1.el7.noarch
foreman-release-scl-7-3.el7.noarch
foreman-selinux-2.1.4-1.el7.noarch
foreman-service-2.1.4-1.el7.noarch
foreman-vmware-2.1.4-1.el7.noarch
gmhrnpefrmn01.esbu.nttict.com.au-foreman-client-1.0-1.noarch
gmhrnpefrmn01.esbu.nttict.com.au-foreman-proxy-1.0-1.noarch
gmhrnpefrmn01.esbu.nttict.com.au-foreman-proxy-client-1.0-1.noarch
rubygem-foreman_maintain-0.6.13-1.el7.noarch
tfm-rubygem-foreman_ansible-5.1.3-1.fm2_1.el7.noarch
tfm-rubygem-foreman_ansible_core-3.0.3-1.fm2_1.el7.noarch
tfm-rubygem-foreman_default_hostgroup-5.0.0-4.fm2_1.el7.noarch
tfm-rubygem-foreman_discovery-16.1.2-1.fm2_1.el7.noarch
tfm-rubygem-foreman_docker-5.0.0-3.fm2_1.el7.noarch
tfm-rubygem-foreman_hooks-0.3.16-2.fm2_1.el7.noarch
tfm-rubygem-foreman_memcache-0.1.1-3.fm2_1.el7.noarch
tfm-rubygem-foreman_openscap-4.0.2-1.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution-3.3.7-1.fm2_1.el7.noarch
tfm-rubygem-foreman_remote_execution_core-1.3.0-1.el7.noarch
tfm-rubygem-foreman_rescue-3.0.0-1.fm2_1.el7.noarch
tfm-rubygem-foreman_setup-7.0.0-2.fm2_1.el7.noarch
tfm-rubygem-foreman-tasks-2.0.2-1.fm2_1.el7.noarch
tfm-rubygem-foreman-tasks-core-0.3.4-1.fm2_1.el7.noarch
tfm-rubygem-foreman_templates-9.0.1-1.fm2_1.el7.noarch
tfm-rubygem-foreman_vault-0.3.0-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman-2.1.3-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_admin-0.0.9-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_admin-doc-0.0.9-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_ansible-0.3.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_ansible-doc-0.3.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_azure_rm-0.2.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_azure_rm-doc-0.2.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_bootdisk-doc-0.3.0-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_discovery-1.0.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_discovery-doc-1.0.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman-doc-2.1.3-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-0.0.5-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_docker-doc-0.0.5-1.el7.noarch
tfm-rubygem-hammer_cli_foreman_kubevirt-0.1.4-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_kubevirt-doc-0.1.4-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_leapp-0.1.0-2.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_leapp-doc-0.1.0-2.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_openscap-0.1.11-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_openscap-doc-0.1.11-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-0.1.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_remote_execution-doc-0.1.2-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_salt-0.0.5-4.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_salt-doc-0.0.5-4.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_ssh-0.0.2-7.el7.noarch
tfm-rubygem-hammer_cli_foreman_ssh-doc-0.0.2-7.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-0.0.14-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_tasks-doc-0.0.14-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_templates-0.2.0-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_templates-doc-0.2.0-1.fm2_1.el7.noarch
tfm-rubygem-hammer_cli_foreman_virt_who_configure-0.0.6-1.el7.noarch

 # hammer repository list | grep -i oel7_
1035 | oel7_latest                | oel7                 | yum          | http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/                       
756  | oel7_optional              | oel7                 | yum          | http://yum.oracle.com/repo/OracleLinux/OL7/optional/archive/x86_64/             
755  | oel7_u9                    | oel7                 | yum          | http://yum.oracle.com/repo/OracleLinux/OL7/9/base/x86_64/                       
1036 | oel7_uek4                  | oel7                 | yum          | http://yum.oracle.com/repo/OracleLinux/OL7/UEKR4/x86_64/                        

hammer repository info --id 1035
ID: 1035
Name: oel7_latest
Label: oel7_latest
Organization: mhr
Red Hat Repository: no
Content Type: yum
Mirror on Sync: yes
URL: http://yum.oracle.com/repo/OracleLinux/OL7/latest/x86_64/
Publish Via HTTP: yes
Published At: http://foremanserver.com.au/pulp/repos/mhr/Library/custom/oel7/oel7_latest/
Relative Path: mhr/Library/custom/oel7/oel7_latest
Download Policy: immediate
Ignorable Content Units: srpm
HTTP Proxy:
ID: 2
Name: proxyserver01
HTTP Proxy Policy: use_selected_http_proxy
Product:
ID: 23
Name: oel7
GPG Key:
ID: 3
Name: RPM-GPG-KEY-oracle-ol7
Sync:
Status: Warning
Last Sync Date: about 4 hours
Created: 2020/11/27 09:03:21
Updated: 2020/11/27 09:03:23
Content Counts:
Packages: 0
Source RPMS: 0
Package Groups: 0
Errata: 0
Module Streams: 0

Hi all,

we are having the same problem with 3.17 and without using a http_proxy.

Susanne

Has anyone been able to rectify this issue yet . As I still have the same problem , this OEL 7 updates channel will not sync no matter what I try. I have removed the optional OEL channel as I had seen in other posts that it may be causing the duplicate package issue. I have also removed the optional repo and run the clean orphans rake job and then tried to resync the repo. It just continuously fails with the following error: Katello::Errors::Pulp3Error: Package matching query does not exist.
This is a fresh centos 7.9 install with foreman 2.3.2, Katello 3.18 with nothing special extra added.

candlepin-3.1.22-1.el7.noarch
foreman-2.3.2-1.el7.noarch
katello-3.18.1-1.el7.noarch
katello-repos-3.18.1-1.el7.noarch
pulp-client-1.0-1.noarch
pulp-katello-1.0.3-1.el7.noarch
pulp-server-2.21.5-1.el7.noarch
qpid-cpp-server-1.39.0-1.el7.x86_64

I have 3 separate foreman / katello installations all with the same setup and they all fail exactly the same way for this repo only.

oel7_latest
Name: oel7_latest
Label: oel7_latest
Description:
Backend Identifier: 5299b55b-829c-49b2-aa45-5b95f3993197
Type: yum

Sync Settings
Restrict to Architecture: No restriction
Restrict to OS version: No restriction
Upstream URL: Oracle Linux 7 (x86_64) Latest | Oracle, Software. Hardware. Complete.
Verify SSL: Yes
Upstream Authorization:

Yum Metadata Checksum: Default
Mirror on Sync: Yes
HTTP Proxy: Use specific HTTP Proxy (proxy01)
Ignorable Content: Source RPM
Publish via HTTPS: Yes
Publish via HTTP: Yes
Published At: http://katello.server.name/pulp/repos/cust1/Library/custom/oel7/oel7_latest/
GPG Key: RPM-GPG-KEY-oracle-ol7
SSL CA Cert:
SSL Client Cert:
SSL Client Key:
Download Policy: Immediate

Sync Status
Sync Interval: Weekly on 2020-12-06 08:00:00 +1100 at 2020-12-06 08:00:00 +1100 (Server Time)
Last Sync: 14 minutes ago (Feb 08, 04:43 PM Local Time)
Next Sync: February 14, 2021, 08:00 AM (Server Time)
Sync State: warning

How can I find out what file is causing the issue or is there some way to make it ignore this one file and continue on downloading the repo ?

Any help at this stage would be great as its a production system and we cant patch the systems without the latest updates being available in this repo.

PS: I also setup a ULN mirror on a seperate OEL 7 machine and mirrored the whole OEL 7 repo from Oracle and tried to import the updates channel using the ULN mirror server as the upstream URL and it still fails with the same error.