[Install Errata] There was an issue with the backend service pulp 500 Internal Server Error

Hi,

When I try to install errata security on RHEL servers, I have the error message in object.

So, I restart katello services. All services work good.

I execute the hammer command : hammer ping
And look the bad result :
candlepin:
Statut: ok
Server Response: Duration: 17ms
candlepin_auth:
Statut: ok
Server Response: Duration: 13ms
pulp:
Statut: FAIL
Server Response:
pulp_auth:
Statut: FAIL
foreman_tasks:
Statut: ok
Server Response: Duration: 2ms

Since, I search where is the pulp problem and don’t find any message which can help me.

Did you have any idea please ?

TCK

PS : sorry for mistakes, I’m french :wink:

Bonjour!

Can you check journalctl and /var/log/messages for pulp errors and paste them here? Also try to restart the pulp services and check for errors.

  sudo systemctl stop pulp_celerybeat.service pulp_workers.service pulp_resource_manager.service
  sudo systemctl start pulp_celerybeat.service pulp_workers.service pulp_resource_manager.service
  sudo systemctl status pulp_celerybeat.service pulp_workers.service pulp_resource_manager.service

Thanks for your answer.

Restart pulp don’t change anything.

In journalctl, I juste have this error (but not pulp) :
déc. 04 15:02:20 katello-02 kernel: httpd[9202]: segfault at 10 ip 00007fedac9c5a8f sp 00007fed921adc30 error 4 in libpython2.7.so.1.0[7fedac963000+17e000]

In /var/log/messages, I have a lot of messages but any error messages. I see one strange thing, I often have this message even I don’t restart pulp:
Dec 4 15:00:43 katello-02 pulp: pulp.server.webservices.application:INFO: *************************************************************
Dec 4 15:00:43 katello-02 pulp: pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized
Dec 4 15:00:43 katello-02 pulp: pulp.server.webservices.application:INFO: *************************************************************

I don’t understand, It’s like pulp restart indefinitely.

FYI :
Katello 3.8

It works, I don’t know why but it works. I don’t understand.

I stop one service which don’t belong to foreman. I will test if it’s that.

1 Like

I have too this critical error :
Jan 24 11:34:29 katello-03 kernel: httpd[1201]: segfault at 10 ip 00007fb2e177309f sp 00007fb2c685fc30 error 4 in libpython2.7.so.1.0[7fb2e1710000+17e000]

I don’t find a lot of things about it. I watch some defunct httpd process.

I have one more stranger thing :
[root@katello-03 ~]# pulp-admin server distributors
There was an internal server error while trying to access the Pulp application.
One possible cause is that the database needs to be migrated to the latest
version. If this is the case, run pulp-manage-db and restart the services. More
information may be found in Apache’s log.

[root@katello-03 ~]# pulp-admin server distributors
^[[A±---------------------------------------------------------------------+
Supported Distributors
±---------------------------------------------------------------------+

Id: yum_clone_distributor
Display Name: Yum Clone Distributor
Types: rpm, srpm, drpm, erratum, distribution, package_category,
package_group
etc…

So when I do many tries, pulp server answer. Maybe, there is something on my server which use same ressources (port ?) than pulp but what ?

Nicolas

Hi,

three months later, I have the same error status.

The Pulp Server seems to restart indefinitely.

When I execute : watch " pulp-admin server distributors", I can see some OK connections. When I look /var/log/messages, I have many messages (about 15 per minutes) :
Apr 10 09:58:09 katello-03 pulp: pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized

My best wai is that U have a problem with a port. I think that 2 processes want access the same port at the same time but how to indentify which port and whoch process ?

I hope to have a reply, I’m on this problem for 3 days, I desesperate, I’m crying…

After a reboot, pulp was ok during some (milli)seconds :
[root@katello-03 ~]# hammer ping
database:
Status: ok
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 22ms
candlepin_auth:
Status: ok
Server Response: Duration: 24ms
pulp:
Status: ok
Server Response: Duration: 573ms
pulp_auth:
Status: FAIL
Server Response: Message: 500 Internal Server Error
foreman_tasks:
Status: FAIL
Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status

[root@katello-03 ~]# hammer ping
database:
Status: ok
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 14ms
candlepin_auth:
Status: ok
Server Response: Duration: 14ms
pulp:
Status: FAIL
Server Response: Message: 500 Internal Server Error
pulp_auth:
Status: FAIL
Server Response: Message: Skipped pulp_auth check after failed pulp check
foreman_tasks:
Status: ok
Server Response: Duration: 3ms

The problem is really about the pulp server. If I wait a (long) moment with the watch command, I can see :
image

So, it works one time during a very short time period. The conncetion with the mongo DB is always reinitialized, infinitely.

A direct connection to DB with mongo command is OK

I’m lost and desesperate

can someone from @Pulp team please take a look at this issue?

Hi,

What helped me in this situation was to run the following command in a root shell:
foreman-installer --scenario katello --upgrade

Regards
Tom

Hi,

thanks for you help but I tried it. I had 3 elements which didn’t upgrade. It’s was the same situation after.
I install a new katello server (it’s OK) but keep the old to test if needed.

TCK

tck-lt were you able to solve it?

ok, from here see section “CAPSULE SERVER”, I was able to run the following to fix the db issue:

$ for i in pulp_celerybeat pulp_resource_manager pulp_workers httpd foreman-proxy; do service $i stop; done
$ sudo -u apache pulp-manage-db
$ for i in pulp_celerybeat pulp_resource_manager pulp_workers httpd foreman-proxy; do service $i start; done

It didn’t fix for me, I have same errors after.

can you verify your mongodb server health? I found out that even after the “fix” I provided the issues were present where pulp would report all was ok but as soon as I did a package task (e.g. sync a repo) it would fail immediately.

I later found out that the issue was when I was trying to move my mongodb folder to another partition. my mv command was not right and had a typo.

sudo -u apache pulp-manage-db “fixed” the error but not the underlying issue.

Hope that helps in your troubleshooting