What I have been doing is deleting all content views that use the repo then remove it from Content > Red Hat Repositories lastly run foreman-rake katello:delete_orphaned_content RAILS_ENV=production >/dev/null 2>&1.
That sounds about right. @katello team may need to pop in and take a look at what could be causing this issue.
Hi @katello folks. I don’t mean to be pushy, but curious if you had thoughts around what I have been experiencing. I have completely removed all RHEL 7 repositories from Katello and started them over several times with the same issue as the outcome. I’m all out of ideas and stuck with no ability to deploy RPMs to RHEL 7 systems via Katello at this time.
Thanks in advance for any suggestions.
I suspect your immediate downloads are failing due to some sort of rate limiting on the server. You can try adjusting the download_concurrency by running:
hammer repository update --id=X --download-concurrency=2
where X is the id of the repository, you could also use --name=REPO_NAME --product=PRODUCT_NAME instead of specifying the id.
And then try re-syncing the repository using the immediate download policy.
As for your installation issues, i’d love to see the output of:
journalctl -f -u pulpcore*
while you try to yum install an rpm.
@Justin_Sherrill thanks for the response and suggestions. I haven’t tried the download_concurrency change and the Red Hat repositories are still using the default “on demand” policy at the moment. Here is the journalctl output while trying to update the sos RPM on a server.
May 25 15:59:09 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:09 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/custom/ORG/ORG_el7_x86_64/repodata/repomd.xml HTTP/1.1” 200 2216 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:10 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:10 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/custom/Extra_Packages_for_Enterprise_Linux/Extra_Packages_for_Enterprise_Linux_7_x86_64/repodata/repomd.xml HTTP/1.1” 200 2552 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:10 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:10 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/custom/Foreman/Foreman_Client_Latest_el7_x86_64/repodata/repomd.xml HTTP/1.1” 200 2519 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:10 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:10 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/custom/Zabbix/Zabbix_5_0_el7_x86_64/repodata/repomd.xml HTTP/1.1” 200 2225 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:11 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:11 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/extras/os/repodata/repomd.xml HTTP/1.1” 200 2547 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:11 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:11 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/optional/os/repodata/repomd.xml HTTP/1.1” 200 2554 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:12 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:11 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/os/repodata/repomd.xml HTTP/1.1” 200 2892 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:12 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:12 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/rhscl/1/os/repodata/repomd.xml HTTP/1.1” 200 2554 “-” “urlgrabber/3.10 yum/3.4.3”
May 25 15:59:16 host.example.com pulpcore-content[1228]: [25/May/2021:20:59:16 +0000] “GET /pulp/content/ORG/Library/rhel-7-server-x86_64-cv/content/dist/rhel/server/7/7Server/x86_64/os/Packages/s/sos-3.9-5.el7_9.4.noarch.rpm HTTP/1.1” 404 172 “-” “urlgrabber/3.10 yum/3.4.3”
Regards,
Cassuis
@Justin_Sherrill
Even with changing the download concurrency the downloading of the RPMs still fails with the policy set to immediate.
@Justin_Sherrill
Not sure I can explain it, but it completed successfully this morning during the normal sync job. I have since changed the download_concurrency back to the default (10) and have successfully done another sync with the policy set to immediate. Once done, I was able to patch some development servers just fine so looks like my issues is fixed.
Thanks to everyone for their input and assistance. I love this product and the community has always been so helpful.
1 Like
Awesome! One thing to note is that if download_concurrency was a problem, the problem may temporarily go away if there’s nothing (or very little) new packages to download. If a new major update is released and 100s of packages are updated, you might hit it again. Keep that in mind.
@Justin_Sherrill
So is the lower the number for download_concurrency better? I would think that the higher the number would be mean better performance.
It does mean better performance but it also means the higher likelyhood that the server will start rejecting your requests. Pulp 2 was a lot more aggressive with retrying downloads. Pulp3 is a lot more conservative in this regard and will fail a sync at the first hint of trouble 
Could you help me with the process on implementing this change on a CentOS 7 installation? I’m not sure exactly where I need to add this option for the ‘Bad Gateway’ error messages.
Thank you.
Which change exactly? the download concurrency change?
Also, i started seeing the same issue with syncs and filed an issue here: Issue #8867: downloads from cdn.redhat fail with 'Cannot connect to host cdn.redhat.com:443 ssl:default [[SSL: SSLV3_ALERT_UNEXPECTED_MESSAGE] sslv3 alert unexpected message (_ssl.c:877)]' - Pulp
@Justin_Sherrill I was looking at the change for implementing the disableuse=no for the pulp core-content service. However I’m having some big issues after applying the updates today. When running any commands to get the status of the services from ‘foreman-maintain service status’ I receive this message.
Running Status Services
Get status of applicable services:
Displaying the following service(s): [FAIL]
undefined method `enabled?’ for RemoteDB(rh-mongodb34-mongod [5]):ForemanMaintain::Utils::Service::RemoteDB
Scenario [Status Services] failed.
The following steps ended up in failing state:
[service-status]
Resolve the failed steps and rerun
the command. In case the failures are false positives,
use --whitelist=“service-status”
Both MongoDB and PostgreSQL are running on a remote server.
hrmm, what version of rubygem-foreman_maintain is installed? sounds like its not taking the ‘remote’ nature into consideration
After updating today it is running rubygem-foreman_maintain-0.7.9-1.el7.noarch. The previous version was rubygem-foreman_maintain-0.7.5-1.el7.noarch. I am able to reproduce in both non-prod and prod environments.
Hi folks!
I faced the same issues on different kind of environment: azure, local kvm… etc just for the sake of testing. I was completely unable to sync any “large” repository… or the sync will just crash at 80% or so…
In the end, it seems this issue disappeared when I gave my VMs a little bit more cpu and memory.
My initial assumption was that those requirements were probably for production grade deployment… I was wrong (and it’s stated in the documentation)… anyway, before going too deep into the code, make sure you meet those 
Any resolution on this? I recently apply updates in our test/qa environment to bring it to 2.3.5 and also migrated the pulpcore database to the environments dedicated postgresql server. Now whenever I try to work with foreman services, I get the error ‘undefined method `enabled?’ for RemoteDB(postgresql:candlepin [10]):ForemanMaintain::Utils::Service::RemoteDB’ for any service related command in foreman-maintain unless I use the switch ‘–exclude postgresql’. I would like to have a solution to this issue before I apply updates and do the same DB migration of pulpcore in our production environment.
Sorry for the late reply, but I haven’t heard of any resolution yet. I still have the same issue with my Katello environment split with the application and data bases on different hosts. I have not upgraded to version 4 yet as I’m still running the latest release of 3.18.
here’s the bug: Bug #32319: service stop with remote databases fails - Foreman Maintain - Foreman
it is resolved but may not be backported to your given release. I would try a
yum update http://yum.theforeman.org/nightly/el7/x86_64/rubygem-foreman_maintain-0.8.9-1.el7.noarch.rpm
that will update to the newest foreman_maintain that has this fix.
1 Like
Also for the download issues, this is resolved in pulp 3.14 by adding retry support . Upgrading to katello 4.1.2 should get that for you.
2 Likes