Problem:
When i used Foreman 2.5 and Katello 4.1.3 i had no memory issue. It was very stable. After i updated Katello 4.1.4 the memory problem began. It went up and i was getting messages: "Linux: OOM killed a process. Out of memory: Kill process 2979 (ruby). So i decided to upgrade Foreman and Katello to the latest version and hoped that it will solve my issues. But unfortunately still memory usage is increasing and its not stable.
Here is a memory usage picture after the upgrade:
There are few changes with respect to increased memory consumption[1]. These are planned for 3.0.1,
As per the discussions of release meetings we are expecting to release 3.0.1 within a few weeks. You can use Release team meeting agenda page to track the discussions about same.
Thanks for the info @upadhyeammit
I have tried to investigate more my memory issue and i checked puma state. It seems, that puma is consumes a quite lot of memory if im correct. Or does it look normal ?
I’ve yet to come across a scenario where a single puma worker has more than 5x the memory consumption of the other ones, so that is interesting.
@kimpsik can you share the output of hammer ping from your server? It contains useful information such as how many events the background workers have handled and I’d like to see if the volume is especially high. Those background workers run only in a single puma process at any given time.
@Jonathon_Turel Here is the requested output of hammer ping:
database:
Status: ok
Server Response: Duration: 0ms
katello_agent:
Status: FAIL
message: Not running
Server Response: Duration: 0ms
candlepin:
Status: ok
Server Response: Duration: 21ms
candlepin_auth:
Status: ok
Server Response: Duration: 17ms
candlepin_events:
Status: ok
message: 59 Processed, 0 Failed
Server Response: Duration: 0ms
katello_events:
Status: ok
message: 2 Processed, 0 Failed
Server Response: Duration: 1ms
pulp3:
Status: ok
Server Response: Duration: 65ms
foreman_tasks:
Status: ok
Server Response: Duration: 24ms
I upgraded to 3.0/4.2.0.1.rc3 just yesterday and today came to find an unresponsive server.
# hammer ping katello
katello_agent:
Status: ok
message: 0 Processed, 0 Failed
Server Response: Duration: 0ms
candlepin:
Status: FAIL
Server Response: Message: Failed to open TCP connection to localhost:23443 (Connection refused - connect(2) for "localhost" port 23443)
candlepin_auth:
Status: FAIL
Server Response: Message: A backend service [ Candlepin ] is unreachable
candlepin_events:
Status: ok
message: 153 Processed, 0 Failed
Server Response: Duration: 0ms
katello_events:
Status: ok
message: 5 Processed, 0 Failed
Server Response: Duration: 0ms
pulp3:
Status: ok
Server Response: Duration: 50ms
foreman_tasks:
Status: ok
Server Response: Duration: 2ms
I did a foreman-maintain service restart and back up and running.
top view sorted on memory…
@kimpsik that looks OK but I am curious about the katello_agent part. Are you using katello-agent to manage packages on your Content Hosts?
If not, I recommend running the installer with --foreman-proxy-content-enable-katello-agent=false to remove the supporting infrastructure from your server.
If you do use it, then something is not right as you can see (this is likely separate from the memory issues). Can you check the status of systemd services qpidd and qdrouterd on your server?
Running the installer to disable the katello-agent support should result in those services being turned off which makes me think the command you ran did not perform as expected. Can you check /etc/foreman/plugins/katello.yaml to see what it says for:
@ehelms My fault. I did copy the services outputs before i ran the command, which will disable the katello-agent support. Now both services cannot be found.
