Foreman 2.0.0 - memory leak?

Support
#21

I was able to build foreman from source (https://github.com/theforeman/foreman.git -b 2.0-stable) with ruby2.7 and added the fix from above.
But now some gems throw a lot of warnings and i’m not sure, if i should bring this in our production environment so close to the weekend.

/opt/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.17/lib/graphql/schema/mutation.rb:70: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.17/lib/graphql/schema/member/has_fields.rb:51: warning: The called method `field' is defined here
/opt/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.17/lib/graphql/schema/mutation.rb:70: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/graphql-1.8.17/lib/graphql/schema/member/has_fields.rb:51: warning: The called method `field' is defined here
/opt/foreman/vendor/ruby/2.7.0/gems/activerecord-5.2.1/lib/active_record/type.rb:27: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/activerecord-5.2.1/lib/active_record/type/adapter_specific_registry.rb:9: warning: The called method `add_modifier' is defined here
/opt/foreman/vendor/ruby/2.7.0/gems/activerecord-5.2.1/lib/active_record/type.rb:27: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/activerecord-5.2.1/lib/active_record/type/adapter_specific_registry.rb:9: warning: The called method `add_modifier' is defined here
/opt/foreman/app/models/concerns/has_many_common.rb:48: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/activerecord-5.2.1/lib/active_record/associations.rb:1368: warning: The called method `has_many' is defined here
/opt/foreman/vendor/ruby/2.7.0/gems/scoped_search-4.1.8/lib/scoped_search/definition.rb:309: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/scoped_search-4.1.8/lib/scoped_search/definition.rb:27: warning: The called method `initialize' is defined here
/opt/foreman/app/models/concerns/has_many_common.rb:48: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/activerecord-5.2.1/lib/active_record/associations.rb:1368: warning: The called method `has_many' is defined here
/opt/foreman/app/models/concerns/has_many_common.rb:53: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/activerecord-5.2.1/lib/active_record/associations.rb:1821: warning: The called method `has_and_belongs_to_many' is defined here
/opt/foreman/app/models/concerns/has_many_common.rb:77: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/activerecord-5.2.1/lib/active_record/associations.rb:1653: warning: The called method `belongs_to' is defined here
/opt/foreman/vendor/ruby/2.7.0/gems/scoped_search-4.1.8/lib/scoped_search/definition.rb:309: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/scoped_search-4.1.8/lib/scoped_search/definition.rb:27: warning: The called method `initialize' is defined here
/opt/foreman/app/models/concerns/has_many_common.rb:53: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/activerecord-5.2.1/lib/active_record/associations.rb:1821: warning: The called method `has_and_belongs_to_many' is defined here
/opt/foreman/app/models/concerns/has_many_common.rb:77: warning: Using the last argument as keyword parameters is deprecated; maybe ** should be added to the call
/opt/foreman/vendor/ruby/2.7.0/gems/activerecord-5.2.1/lib/active_record/associations.rb:1653: warning: The called method `belongs_to' is defined here
1 Like
#22

SHIP IT! :slight_smile:

We have a lots of deps, chances are this might break. Wait for Monday morning then.

#23

Yes, i will wait for Monday.

Your fix has a very good impact on CPU consumption:
image

1 Like
#24

Wow you run Foreman with 10g Memory? i pimped by Foreman Server with 6 Core and 32GB Ram for around 100 Systems. Every 2-3 Days the System explodes eating all Memory and Swapspace. Most times mongod get killed because out of memory. Sometimes Publishing a large content view (Ol7, Epel some addtional small repos) lead to of of memory and i have to restart Foreman. I have this issue with two Server over all Foreman Versions the last 2 Years.

1 Like
#25

Seems to be something really weird at your configuration. We’ve connected around 2000 Systems to our production system (4x 4CPUs+ 10G Ram), but no plugins added yet :wink:

#26

I think it’s apples and oranges here. @chr1s is running pure Foreman (from sources) without Katello (perhaps even Puppet on a different node?) while @sjansen appears to run Katello-MongoDB-Candlepin-Puppet full stack.

Anyway, that patch did help for CPU a little bit. I guess you don’t see any memory improvements.

Any news to share?

#27

One sidenote, @ekohl have realized that Puppet is calling ENC twice for some reason. If we find why is that and fix this, 1/2 of your queries would disappear. This won’t help with memory hog much, possibly makes it slower but it is worth noting.

#28

Yes, thank you for reminding me. It looks like I haven’t created a ticket for the double ENC call. That’s entirely done by Puppetserver because even with a trivial ENC script, it’s called twice.

#29

https://tickets.puppetlabs.com/browse/SERVER-2818

#30

Sorry for the late reply. I forgot we had a holiday on Monday :slightly_smiling_face:
Last week I tried different configurations of puma (worker/threads/…) It seems that the following configurations works fine for me (after the fix and i don’t know what it would look like without it):
Worker 4, threads 4,8 @ruby2.5
image

Foreman with ruby2.7
Since the configuration now works fine and I’m still getting a lot of warning with ruby2.7, i dont know, if i should bring this packages in our production. What is your experience using ruby2.7?

#31

At this moment we don’t have any supported platform on Ruby 2.7. @mmoll is working on Ubuntu 20.04 support (which ships 2.7) but I don’t know if specific 2.7 fixes were needed.

#32

To me it looks very similar to what you had with Ruby 2.5, 4 workers hitting almost 8 gigs which is 2 GB per worker. We have seen our process to peak at 2 GB and that should be probably the treshold when we should set up automatic worker restart with a tool like: https://github.com/schneems/puma_worker_killer

The only difference is that we have seen this with Foreman + many plugins installed. You say you have just Foreman core. That’s weird. Maybe ton of ENC data? Dunno.

#33

Remember the first post i’ve created. The puma configuration was workers: 2, threads 0,16 (according to your defaults from the puppet modul) and the memory consumption:
foreman_memory_consumption.

Perhaps threads cosume more memory or is this a problem of ruby2.5 memory fragmentation?!

#34

Ok thanks for comparsion, that’s like 20% reduction. Not bad.

Yeah it really looks like the Ruby 2.7 memory de-fragmentation really helps here. But still, even if there was no Copy on Write what we are seing here is 4 GB per one worker. That’s way too much. There must be a memory leak somewhere.

Does dtrace, systemtap or the modern bpftrace work on your distro?

Can you share how many CPUs/cores do you have? It is worth looking into MALLOC_ARENA_MAX setting, currently you probably use the default value which is often a bit too high making the best multithreaded performance at the cost of some memory. Try to decrease this to just 2. For more info: https://developers.redhat.com/blog/2017/03/02/malloc-internals-and-you/

#35

Not that we misunderstand each other, the differences only come from changing the puma settings from workers:2, threads:0,16 to workers:4, threads:4,8 + your externalnode fix. We’re still using ruby2.5

Systemtap is available on SLES12SP5, but i dodnt know how to use it…I’ve never done anything like this before. If you tell me exactly what you want me to do, we can try this.

We’re using virtual machines (VMWare), each virtual machine has 4 virtual CPUs (system hardware based on Intel® Xeon® CPU E5-2699 v3 @ 2.30GHz).

#36

You can try to play around with https://github.com/lzap/foreman-tracer - a set of SystemTap scripts that can help tracking issues. For example foreman-tracer rails objects-total should be particularly interesting. You need to have Ruby compiled with STAP/DTRACE statements, which is by default for modern Ruby versions. But if it does not work for you, don’t loose much time for this.

I suggest trying setting this in foreman.service:

Environment=MALLOC_ARENA_MAX=2

We actually do this for dynflow processes and I wonder why we don’t do the same since we default up to 16 threads now. @ekohl do you think this is worth adding into 2.1?

#37

I think that makes sense. Should be as easy as changing https://github.com/theforeman/foreman/blob/develop/extras/systemd/foreman.service. That would have been more complicated with Passenger.

Foreman 2.1 test week
#38
#39

I tried to run foreman-tracer, but some binaries were missing. Scl was the last, before i decided to test MALLOC_ARENA_MAX=2 first.
“foreman.service” is corrected and now we need to wait :slight_smile:

1 Like
#40

Here you can see the graphs after almost one day. (Puma config: worker=4, threads=4,8)

Node1 with MALLOC_ARENA_MAX=2
image

Node2 without MALLOC_ARENA_MAX
image

3 Likes