Unable to update manifest on Satellite 6.4.4.2 - 500 Server Error

Problem:

I have a Red Hat Satellite server (apologies if posting for this is not allowed here - however a ticket had been open with Red Hat for >30 days with no resolution in sight and I hoped there might be some experts here who could assist). Satellite version is 6.4.4.2 (appears to be Foreman 1.18 with Katello 3.7.0). We are unable to either refresh the subscription manifest or re-upload the manifest. Both tasks hang at 1% for about 30 minutes, and then either timeout or 500 Internal Server Error’s are logged against the task. Various attempts to resolve this through patching, clearing out old, stuck tasks and re-performing, and so on have all come to nothing.

Expected outcome:

Should be able to refresh manifest and upload manifest.

Foreman and Proxy versions:

Foreman: 1.18
Proxy: 1.18

Foreman and Proxy plugin versions:

Distribution and version:

RHEL 7.8

Other relevant data:

I have lots of logs, task exports, sosreports and so on that have all been historically sent to Red Hat support. Happy to upload or provide links to but didn’t want to bombard on my first post. Not sure how to debug this issue at this stage - any and all guidance gratefully appreciated.

Hi @jamesf

First let’s try katello-service restart; then hammer ping. Everything working ok?

Does the problem persist if you download a new manifest and try it instead?

Also, do you have the output of /var/log/foreman/production.log immediately after a failed manifest import?

Hi @jeremylenz

Thanks so much for replying. In answer to your question - katello-service restart works fine and all services report that they stop and start cleanly. hammer ping also works fine.

I’ve tried downloading a new manifest and uploading it - that results in (what looks from the outside) like the same issue - it hangs for ages then reports 500 internal server errors. I’ll get the output from production.log for you after a failed manifest import and post to this thread as soon as I can.

Thanks again!

James