[Install Errata] There was an issue with the backend service pulp 500 Internal Server Error

Support
1

Hi,

When I try to install errata security on RHEL servers, I have the error message in object.

So, I restart katello services. All services work good.

I execute the hammer command : hammer ping
And look the bad result :
candlepin:
Statut: ok
Server Response: Duration: 17ms
candlepin_auth:
Statut: ok
Server Response: Duration: 13ms
pulp:
Statut: FAIL
Server Response:
pulp_auth:
Statut: FAIL
foreman_tasks:
Statut: ok
Server Response: Duration: 2ms

Since, I search where is the pulp problem and don’t find any message which can help me.

Did you have any idea please ?

TCK

PS : sorry for mistakes, I’m french :wink:

2

Bonjour!

Can you check journalctl and /var/log/messages for pulp errors and paste them here? Also try to restart the pulp services and check for errors.

  sudo systemctl stop pulp_celerybeat.service pulp_workers.service pulp_resource_manager.service
  sudo systemctl start pulp_celerybeat.service pulp_workers.service pulp_resource_manager.service
  sudo systemctl status pulp_celerybeat.service pulp_workers.service pulp_resource_manager.service
3

Thanks for your answer.

Restart pulp don’t change anything.

In journalctl, I juste have this error (but not pulp) :
déc. 04 15:02:20 katello-02 kernel: httpd[9202]: segfault at 10 ip 00007fedac9c5a8f sp 00007fed921adc30 error 4 in libpython2.7.so.1.0[7fedac963000+17e000]

In /var/log/messages, I have a lot of messages but any error messages. I see one strange thing, I often have this message even I don’t restart pulp:
Dec 4 15:00:43 katello-02 pulp: pulp.server.webservices.application:INFO: *************************************************************
Dec 4 15:00:43 katello-02 pulp: pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized
Dec 4 15:00:43 katello-02 pulp: pulp.server.webservices.application:INFO: *************************************************************

I don’t understand, It’s like pulp restart indefinitely.

FYI :
Katello 3.8

4

It works, I don’t know why but it works. I don’t understand.

I stop one service which don’t belong to foreman. I will test if it’s that.

1 Like
7

I have too this critical error :
Jan 24 11:34:29 katello-03 kernel: httpd[1201]: segfault at 10 ip 00007fb2e177309f sp 00007fb2c685fc30 error 4 in libpython2.7.so.1.0[7fb2e1710000+17e000]

I don’t find a lot of things about it. I watch some defunct httpd process.

8

I have one more stranger thing :
[root@katello-03 ~]# pulp-admin server distributors
There was an internal server error while trying to access the Pulp application.
One possible cause is that the database needs to be migrated to the latest
version. If this is the case, run pulp-manage-db and restart the services. More
information may be found in Apache’s log.

[root@katello-03 ~]# pulp-admin server distributors
^[[A±---------------------------------------------------------------------+
Supported Distributors
±---------------------------------------------------------------------+

Id: yum_clone_distributor
Display Name: Yum Clone Distributor
Types: rpm, srpm, drpm, erratum, distribution, package_category,
package_group
etc


So when I do many tries, pulp server answer. Maybe, there is something on my server which use same ressources (port ?) than pulp but what ?

Nicolas

12

Hi,

three months later, I have the same error status.

The Pulp Server seems to restart indefinitely.

When I execute : watch " pulp-admin server distributors", I can see some OK connections. When I look /var/log/messages, I have many messages (about 15 per minutes) :
Apr 10 09:58:09 katello-03 pulp: pulp.server.webservices.application:INFO: The Pulp server has been successfully initialized

My best wai is that U have a problem with a port. I think that 2 processes want access the same port at the same time but how to indentify which port and whoch process ?

I hope to have a reply, I’m on this problem for 3 days, I desesperate, I’m crying


14

After a reboot, pulp was ok during some (milli)seconds :
[root@katello-03 ~]# hammer ping
database:
Status: ok
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 22ms
candlepin_auth:
Status: ok
Server Response: Duration: 24ms
pulp:
Status: ok
Server Response: Duration: 573ms
pulp_auth:
Status: FAIL
Server Response: Message: 500 Internal Server Error
foreman_tasks:
Status: FAIL
Server Response: Message: some executors are not responding, check /foreman_tasks/dynflow/status

[root@katello-03 ~]# hammer ping
database:
Status: ok
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 14ms
candlepin_auth:
Status: ok
Server Response: Duration: 14ms
pulp:
Status: FAIL
Server Response: Message: 500 Internal Server Error
pulp_auth:
Status: FAIL
Server Response: Message: Skipped pulp_auth check after failed pulp check
foreman_tasks:
Status: ok
Server Response: Duration: 3ms

16

The problem is really about the pulp server. If I wait a (long) moment with the watch command, I can see :
image

So, it works one time during a very short time period. The conncetion with the mongo DB is always reinitialized, infinitely.

17

A direct connection to DB with mongo command is OK

I’m lost and desesperate

18

can someone from @Pulp team please take a look at this issue?

19

Hi,

What helped me in this situation was to run the following command in a root shell:
foreman-installer --scenario katello --upgrade

Regards
Tom

1 Like
20

Hi,

thanks for you help but I tried it. I had 3 elements which didn’t upgrade. It’s was the same situation after.
I install a new katello server (it’s OK) but keep the old to test if needed.

TCK

21

tck-lt were you able to solve it?

22

ok, from here see section “CAPSULE SERVER”, I was able to run the following to fix the db issue:

$ for i in pulp_celerybeat pulp_resource_manager pulp_workers httpd foreman-proxy; do service $i stop; done
$ sudo -u apache pulp-manage-db
$ for i in pulp_celerybeat pulp_resource_manager pulp_workers httpd foreman-proxy; do service $i start; done
23

It didn’t fix for me, I have same errors after.

24

can you verify your mongodb server health? I found out that even after the “fix” I provided the issues were present where pulp would report all was ok but as soon as I did a package task (e.g. sync a repo) it would fail immediately.

I later found out that the issue was when I was trying to move my mongodb folder to another partition. my mv command was not right and had a typo.

sudo -u apache pulp-manage-db “fixed” the error but not the underlying issue.

Hope that helps in your troubleshooting

25

Hello,

I’m desesperate. I reproduce the same error on my last Katello Server but I know how to reproduce it.
I execute this command :
liste_errata=hammer erratum list --errata-restrict-installable true --organization "CD35" | grep "| security |" | grep RHSA | awk '{printf $3","}' | sed "s/.$//"
hammer host-collection erratum install --errata=$liste_errata --name RHEL_TEST --organization “CD35”

Since the firts execution of this script, Pulp server is down and even a restart doesn’t worked it.

In /var/log/messages, logs continue to display each 30 seconds like in this video (at 13" and 43") :
https://youtu.be/mXG91meks60

Please help me.

26

Well I looked at your logs and it is exactly what I was mentioning. can you check the health of the mongodb server?

eg. stop all services:
foreman-maintain service stop

then start them back with
foreman-maintain service start

see the output.

Then check for the mongodb status
systemctl status rh-mongodb34-mongod

If the above is good then confirm from the UI:
Got to Administer->About

Check the proxy status and in the bottom check the Backend System Status

27

status mongodb OK but pulp down

screenshot_102
screenshot_102885×64 9.25 KB
screenshot_103
screenshot_103897×271 28.2 KB
screenshot_104
screenshot_1041671×643 181 KB