Foreman and smart proxy servers are getting into hung state very frequently

Support
1

**Problem:foreman and smart proxy servers are getting into hung state very frequently

**Expected outcome:Servers to be online

**Foreman and Proxy versions:3.3.1

**Foreman and Proxy plugin versions:3.3.1

**Distribution and version:CentOS 8 Stream running in kernel 4.18.0-448.el8.x86_64

Other relevant data:

2

servers are getting hung frequently post OS upgrade from CentOS 7.9 to CentOS 8 stream. Is there anything to be done post the upgrade.

3

I’m getting an error “We’re sorry, but somethings went wrong” when access foreman console, i tried to ssh foreman server but not able to login, when i access the VM console, i’m not able to enter password to login, so had to reset/power off and power on the VM to bring up foreman console, this is happening frequently just after the OS upgrade happened couple of days back.

I checked VM and server logs as well as foreman production log, i don’t see the reason, is there any other logs to check for the cause?

4

can anyone assist on this? I’m seeing same behavior in one of the smart proxy server as well.

5

i configured kdump in the servers and found cpu load gone to high and systemd in ‘D’ state and saw couple of processes were in zombie state. Couldn’t find which processes were in zombie state due to the ssh connection was establishing.

Today i upgraded foreman and smart proxy servers from 3.3 to 3.4, i still see foreman server and one of smart proxy server (as of now) gone to unresponsive state, in top command status i saw cpu load was at 73 which is too high and systemd was in ‘D’ state, i had to reboot the server to bring back foreman console.

I switched kernel to 3.xx from 4.xx, it’s working fine but foreman and smart proxy servers are going to hung state when running in 4.xx kernel version due to systemd in D state, some zombie processes and high cpu load.

Can i get some help on this to fix?

6

I recommend redeploy Foreman from scratch on 8. When stuff goes bad after a CentOS 7 to 8 migration, it can be pretty hard to pinpoint what the problem is unless you are a Linux expert.

7

how do i migrate all organization, products, hosts, etc from current foreman to new foreman?