Memory consumption increases during GET with smart-proxy possible leak

Support
1

Problem:
Memory consumption increases during GET operation and is not released after it is completed.
I, [2019-01-23T10:04:24.884357 ] INFO – : 10.15.0.231 - - [23/Jan/2019:10:04:24 +0000] “GET /puppet/environments/ITOP2242/classes HTTP/1.1” 200 212982 34.8761
I, [2019-01-23T10:04:25.116733 ] INFO – : 10.15.0.231 - - [23/Jan/2019:10:04:25 +0000] “GET /puppet/environments/ITOP2242/classes HTTP/1.1” 200 212982 95.3287

Expected outcome:
Memory is released

Foreman and Proxy versions:
Foreman

System Information

Version 1.15.6 © 2009-2019 Paul Kelly and Ohad Levy
proxy
vmkhpuppet04 Puppet and Logs 1.15.6
Foreman and Proxy plugin versions:
not sure
Other relevant data:
There are no errors in the log. I cannot send you complete log as per company security policy.
Those are operations during which memory consumption increases:
/var/log/foreman-proxy/proxy.log
I, [2019-01-23T09:33:07.356199 ] INFO – : Successfully initialized ‘foreman_proxy’
I, [2019-01-23T09:33:07.357593 ] INFO – : Successfully initialized ‘puppet_proxy_legacy’
I, [2019-01-23T09:33:07.357669 ] INFO – : Successfully initialized ‘puppet_proxy_ssh’
I, [2019-01-23T09:33:07.357709 ] INFO – : Successfully initialized ‘puppet’
I, [2019-01-23T09:33:07.357751 ] INFO – : Successfully initialized ‘logs’
I, [2019-01-23T09:33:07.360426 ] INFO – : WEBrick 1.3.1
I, [2019-01-23T09:33:07.360510 ] INFO – : ruby 2.0.0 (2015-12-16) [x86_64-linux]
…
I, [2019-01-23T09:33:07.363500 ] INFO – : WEBrick::HTTPServer#start: pid=35259 port=8443
…
I, [2019-01-23T10:04:24.884357 ] INFO – : 10.15.0.231 - - [23/Jan/2019:10:04:24 +0000] “GET /puppet/environments/ITOP2242/classes HTTP/1.1” 200 212982 34.8761
I, [2019-01-23T10:04:25.116733 ] INFO – : 10.15.0.231 - - [23/Jan/2019:10:04:25 +0000] “GET /puppet/environments/ITOP2242/classes HTTP/1.1” 200 212982 95.3287

[root@hostname# uname -a
Linux hostname 3.10.0-957.1.3.el7.x86_64 #1 SMP Thu Nov 29 14:49:43 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[e.g. logs from Foreman and/or the Proxy, modified templates, commands issued, etc]
(for logs, surround with three back-ticks to get proper formatting, e.g.)

logs
2

Upgrade to the latest version to see if the problem persists.

Ruby never returns memory back to the OS, but eventually GC should collect and claim some memory. Can you put memory consumption into some monitoring and get out graph out of it? Does the consumption stabilize at some point (normal) or does it grow forever (abnormal)?

3

Also recent versions of proxy have modular puppet retrievers, which one do you test with? I think they are named V2 and V3.

4

Many thanks for reply. Will try to get a graph out today.
Ultimately we are looking at upgrading and I was just wondering if anyone has seen this before.
Currently we are controlling it with foreman-proxy service restarts.
About the modular retrievers - where can I find them?

5

The feature I was referring to is: Feature #15095: Add Puppet server environment_classes API support - Smart Proxy - Foreman

Depending on Puppet Master version, we use two different implementations. One implementation does use some kind of cache, search for *_cache attributes. These seem to be hash and not LRU, so chances are if you have many classes which are changing those hashes are increasing incrementally eating lot of memory. This is my wild guess.

Try this:

6

from puppet_proxy_legacy.yml

Enable/disable puppet class cache

Cache options

#:use_cache: true

As systems are live I did not have a chance to simulate the problem but I will try to build small lab.
Many thanks again I will have a look at the links.

7

it seems that we are using V3
[root@hostname puppet_proxy_puppet_api]# ls -ltr
total 32
-rw-r–r--. 1 root root 378 Oct 10 2017 v3_environments_retriever.rb
-rw-r–r--. 1 root root 4879 Oct 10 2017 v3_environment_classes_api_classes_retriever.rb
-rw-r–r--. 1 root root 1424 Oct 10 2017 v3_classes_retriever.rb
-rw-r–r--. 1 root root 1393 Oct 10 2017 v3_api_request.rb

8

Can you hotfix the file (see the PR) and restart the process to see how many entries are in cache over time?

9

Hi,
Many thanks for the effort.
Unfortunately it is live environment and it takes ages to approve the hot fix. I will try.

Cheers,
Alex

10

I have improved the patch, while I was working on the debug logging I noticed that entries are not actually removed from cache hash, so refactored this to delete. This could help in some cases.

1 Like
11

I realized there is an attempt to implement proper LRU cache for the importer. Maybe you want to try it?