Problem:
some packages report Status code: 500 when trying to download and install
Expected outcome:
packages to be downloaded and installed.
Foreman and Proxy versions:
foreman 3.10/katello 4.12.1
Distribution and version:
RHEL 8.10
Other relevant data:
This is a fresh install of foreman on a RHEL8 box. host is rhel9.3. some, but not all content downloads and installs successfully. How do I go about troubleshooting why the host is seeing a “Status code: 500” when trying to install packages?
On the server, in /var/log/foreman/production.log you should see something like Completed 500 Internal Server Error at the time the host made the request. There will be a long error traceback just above it - can you please post that here?
I checked both production.log and production.log-#### and neither show “Completed 500” or “internal”
here’s what I am seeing on at least two hosts for this specific package:
yum install gdb
Updating Subscription Management repositories.
Red Hat Enterprise Linux 9 for x86_64 - BaseOS (RPMs) 59 kB/s | 4.1 kB 00:00
Red Hat Enterprise Linux 9 for x86_64 - AppStream (RPMs) 73 kB/s | 4.5 kB 00:00
Red Hat CodeReady Linux Builder for RHEL 9 x86_64 (RPMs) 66 kB/s | 4.5 kB 00:00
Dependencies resolved.
=================================================================================================================================================================================================================
Package Architecture Version Repository Size
=================================================================================================================================================================================================================
Installing:
gdb x86_64 10.2-13.el9 rhel-9-for-x86_64-appstream-rpms 123 k
Installing dependencies:
boost-regex x86_64 1.75.0-8.el9 rhel-9-for-x86_64-appstream-rpms 279 k
gdb-headless x86_64 10.2-13.el9 rhel-9-for-x86_64-appstream-rpms 3.9 M
libbabeltrace x86_64 1.5.8-10.el9 rhel-9-for-x86_64-appstream-rpms 194 k
libipt x86_64 2.0.4-5.el9 rhel-9-for-x86_64-appstream-rpms 58 k
source-highlight x86_64 3.1.9-11.el9 rhel-9-for-x86_64-appstream-rpms 685 k
Transaction Summary
=================================================================================================================================================================================================================
Install 6 Packages
Total download size: 5.2 M
Installed size: 17 M
Is this ok [y/N]: y
Downloading Packages:
[MIRROR] boost-regex-1.75.0-8.el9.x86_64.rpm: Status code: 500 for https://redacted/pulp/content/emus/Library/content/dist/rhel9/9/x86_64/appstream/os/Packages/b/boost-regex-1.75.0-8.el9.x86_64.rpm (IP: 10.110.81.77)
[MIRROR] libipt-2.0.4-5.el9.x86_64.rpm: Status code: 500 for https://redacted/pulp/content/emus/Library/content/dist/rhel9/9/x86_64/appstream/os/Packages/l/libipt-2.0.4-5.el9.x86_64.rpm (IP: 10.110.81.77)
[MIRROR] source-highlight-3.1.9-11.el9.x86_64.rpm: Status code: 500 for https://redacted/pulp/content/emus/Library/content/dist/rhel9/9/x86_64/appstream/os/Packages/s/source-highlight-3.1.9-11.el9.x86_64.rpm (IP: 10.110.81.77)
[MIRROR] boost-regex-1.75.0-8.el9.x86_64.rpm: Status code: 500 for https://redacted/pulp/content/emus/Library/content/dist/rhel9/9/x86_64/appstream/os/Packages/b/boost-regex-1.75.0-8.el9.x86_64.rpm (IP: 10.110.81.77)
[FAILED] boost-regex-1.75.0-8.el9.x86_64.rpm: No more mirrors to try - All mirrors were already tried without success
(2-3/6): source-highlight-3.1.9-11.el9.x86_64.rpm 5% [==== ] 57 kB/s | 279 kB 01:29 ETA
The downloaded packages were saved in cache until the next successful transaction.
You can remove cached packages by executing 'yum clean packages'.
Error: Error downloading packages:
boost-regex-1.75.0-8.el9.x86_64: Cannot download, all mirrors were already tried without success
I have had this same problem - I get 500 errors on some packages, with content views that have not changed.
Nothing in the production log. the http log shows a 500 error, but no detail.
I have removed/re-added the repo from the content view, published a new version, deleted the package and re-sync’d the repo, disabled/re-enabled the repo from the client side. Nothing seems to update the view from an existing view. I have actually removed the repo from my content entirely, and resync’d it and created a new repo, and that seems to have fixed it, but it seems to have happened again.
I am on 3.10 and katello 4.12.1
RockyLinux8 host with a Rocky 9 Client
What version of Candlepin?
Upon further troubleshooting, I think my issue is totally unrelated, and is a filesystem issue rather than a software issue. I have a broken symlink from /var/lib/pulp on my system, and a bunch of stuff got over-written. Fallacy of not sizing my /var appropriately. I think i need to totally redo all my repos, with a better mount point. Thanks for the quick response, and I’ll let you know if i need further assistance.
hmm, then I don’t think you should have the issue present in Candlepin 4.4.10…
On the server, is there anything interesting in /var/log/candlepin/error.log ?
this seems to be reported a few times. it looks like the log has rotated since I last tried to install gdb.
2024-08-13 13:35:01,046 [thread=Thread-5 (org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnector)] [=, org=, csid=] WARN org.apache.activemq.artemis.utils.actors.OrderedExecutor - Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@3045cd6c
java.lang.IllegalStateException: Failed to write to handler on connector org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1@3045cd6c
at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:219)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:57)
at org.apache.activemq.artemis.utils.actors.OrderedExecutor.doTask(OrderedExecutor.java:32)
at org.apache.activemq.artemis.utils.actors.ProcessorBase.executePendingTasks(ProcessorBase.java:68)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at org.apache.activemq.artemis.utils.ActiveMQThreadFactory$1.run(ActiveMQThreadFactory.java:118)
Caused by: java.util.concurrent.RejectedExecutionException: Task org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler$1@507f9641 rejected from org.apache.activemq.artemis.utils.ActiveMQThreadPoolExecutor@496fd39f[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 16]
at java.base/java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2065)
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:833)
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1365)
at org.apache.activemq.artemis.core.client.impl.ClientSessionFactoryImpl$DelegatingBufferHandler.bufferReceived(ClientSessionFactoryImpl.java:1358)
at org.apache.activemq.artemis.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:211)
... 6 common frames omitted
Ok my issue is still happening after rebuilding my repos. My repos are ‘syncing’ successfully but they do not download any data.
The repo information is all present, and it looks like it should be working, but no data is present on disk. it seems like it is getting the metadata but nothing else.
That’s normal for On-demand repos - they only download metadata. You can try setting a repo’s download policy to Immediate, if you want it to download everything. Could also try a “Complete sync” of the repo? but I’m not sure this is a Pulp problem yet…
I’ve done a complete sync, and it still does not bring back any content. I am currently doing a content checksum on all the repos. It seems every package is being identified as missing or repaired as corrupted. This is affecting all the repos i’ve created in the last 24 hours.
Perhaps a manifest refresh will help?
The content checksum seems to fix the issues, at least for now. It forces the packages to download at least.
1 Like
