[Install Errata] There was an issue with the backend service pulp 500 Internal Server Error

tck-lt · June 10, 2020, 12:13pm

You can look at the pulp-admin server status in this video :
https://youtu.be/F_HSf7ncZJY

I don’t know why the connection between the mongo DB and pulp server fail serveral times in time intervall.
The mongodb status is always OK.

Dennis_Kliban · June 10, 2020, 12:56pm

Are you still seeing a segfault in httpd as you were seeing on Jan 24?

tck-lt · June 10, 2020, 1:10pm

Yes

mcorr · June 10, 2020, 1:44pm

Can you ensure that it is not an environmental issue by updating httpd or the wsgi module for it?

If you’re experiencing segfaults, you might need to contact RH customer support to figure out what is happening in your RHEL environment.

tck-lt · June 10, 2020, 2:07pm

I check logs and see that this error appear only juste after the errata security install.
Jobs of install finished at 11h27’38", httpd segfault begin at 11h29’41"

I think if I solve this error, pulp will work but how do it…

NB : httpd and mod_wsgi are already installed in their last version

Thansk a lot.

tck-lt · June 10, 2020, 2:13pm

It means like that https://access.redhat.com/solutions/2429671

(I’m french !)

Cause

Having a client machine in locale using nonASCII characters, such that some yum or goferd error message can contain some of them. This can happen when e.g. trying to install a package to a system with French locale and full disk. A message with error string containing nonASCII character is then sent by goferd into pulp.task qpid queue.

When wsgi:pulp script fetches such a message, it tries to convert it to (ASCII) string (using str python method), what raises exception / segfault.

Since the message isn’t deleted from the queue, it remains there and the wsgi script reads it again just after its start-up.

tck-lt · June 10, 2020, 2:19pm

So, I think this is the cause but I haven’t the solution to avoid the problem. Maybe change my local parameters on my katello server ?

tck-lt · June 10, 2020, 2:28pm

Edit : My answer is not exact. The problem of local is on the client server. So, I must upgrade goferd on each server to see or check install errata on each server one by one

besmirzanaj · June 10, 2020, 4:38pm

at this point I think your mongodb database is busted. You should probably start from scratch and re-enroll your clients

tck-lt · June 11, 2020, 5:43am

No,I won’t. It’s the third time that I install a new Katello from scratch and each time, problem appear when I install errata. It will be not better.

besmirzanaj · June 11, 2020, 5:51am

What do you mean by: install errata?

tck-lt · June 11, 2020, 6:17am

When I execute : hammer host-collection erratum install --errata

tck-lt · June 11, 2020, 7:45am

The problem is only with RHEL and CentOS 6. With RHEL and CentOS 7, the problem doesn’t appear.

tck-lt · June 11, 2020, 3:10pm

Alleluia !!!

I found the problematic client server. When I delete it from the host-collection, I haven’t the problem. When I try an errata install on it, I have this message : Error: Host did not respond within 20 seconds. The task has been cancelled. Is katello-agent installed and goferd running on the Host?

I had a configuration virt-who error on this server (I forgot to comment old lines). It works now for install errata on this server. I try to add again this server in the host collection and restart the group install. It works !

Thansk all for help !

The command who save me to stop the 500 pulp error is :
qpid-config del queue pulp.task --ssl-certificate=/etc/pki/pulp/qpid/client.crt -b amqps://localhost:5671 --force

Restart server don’t work to reset taskes, you must use this command if you have the same error than me

I can close ticket !