Hello everybody,
this weekend we migrated from Foreman 2.0 to Foreman 2.2.2. After the migration the edit page for “Puppet Classes” and “Smart Class Parameters” takes up to 5 minutes to open in Chrome.
Our environment consists of 5 environments, ~600 (sub-)class, ~4700 smart class parameters and ~ 1800 hosts. We use Foreman as ENC, so there are a lot of overrides.
Foreman is running on 4 nodes, each with 4CPUs and 10G Memory
Are there any known issues? What kind of data do you need to help me solving this performance problem?
Additional information:
I opened the devel tools in chrome and started to record the performance of an average puppet modul (average == configured on 50% of the hosts, ~40 smart class parameters). I know recording always slows down the requests//chrome, but perhaps it can give a hint
small extract:
Can you try turning on sql logging (in /etc/foreman/settings.yaml and restart) and share the request output from /var/log/foreman/production.log? be sure to turn off sql logging afterwards as it will quickly fill up your log files.
Looking at the logs, it appears you were right - the main delay is in the front-end, not the server (the longer one took ~5 seconds to load on the server side, which is not very fast but makes sense when there are thousands of elements to render, since there is no pagination for the parameters). However, as far as i could tell there haven’t been any significant changes on this page in years so it’s odd it changed so much between 2.0 and 2.2. since this is a client-side issue, can you try checking the page from a different computer/browser and see if the slowness is still present?
Thousans of overrides sounds a little bit to much . The Puppetclass is assigned to ~ 1800 hosts and has 27 Smart Class Parameters. Thereof approx. the half of the parameters are enabled for override. I think the total count of matchers results in 1000-1500 overrides.
I tried this on serveral other computers and browsers (firefox,chrome,edge), the result is always the same.
The main problem seems to be in the “activateTooltips” function in the java script stack. After deactivating (through adding “return;” as the first statement in the function) we could reduce the time from ~5min to ~45s.
As we need to build foreman from source (Foreman :: Manual) are there some dependencies/libaries (npm,…) which need to be set to an specific version?
If i’m not mistaken, we’re building with node 10 and npm 6, but I’m not sure if that matters in this case, as it looks like the function that generates tooltips is working too hard.
Looking at the overrides part of the page, looks like each override includes two tooltips - one on the fullscreen button and another on the delete button. However these have been there since at least 1.16, so I’m not sure why you would see the slowness only when upgrading from 2.0 to 2.2?
The problem seems also to be in the official packages. I installed Foreman on a Rhel7.6 (from the repo Index of /releases/2.2/el7/x86_64) and connected this instance to our Postgres DB.
Time to load the pages are almost the same. So building foreman with node10/npm6 or node12/npm12 seems not to be the problem.
Thats a good question, hopefully someone can find an answer to this?
I’m not sure if we could solve it in a general way without reworking the whole page or the way we display tooltips - we don’t often have pages that require displaying thousands of tooltips.
As an option that may help, perhaps try modifying this line:
There is only one Parameter with ~1800 overrides. In my opinion this is a range of overrides, which should be expected in some bigger environments (if you use Foreman as ENC).
And with Foreman 2.0 everything was ok. It wasnt the fastest, but “only” took ~8-12s to load this modul.
So there really was no changes which could result in this behaviour?
I dont want to appear impolite, but i need to ask. Do you recommend us to stay with foreman as enc in this scale (number of host, puppetclasses, overrides)?
You need to restart the server for the change to take affect.
As I mentioned, this is quite odd, as this page had no changes between 2.0 and 2.2 - so it’s quite difficult to understand why all of a sudden it became so slow
I believe the ENC functionality should still be fine, it’s only this page load that is problematic. An alternative option is to edit the parameter overrides using the API or CLI. Also, you can edit the specific parameters directly from Configure -> Smart Class Parameters menu, I would expect that all but the one that has 1852 overrides would load quite quickly (I tested locally with 150 overrides which seems to load with no issue). How long does it take to load that one parameter directly? Also, perhaps if practically every host has its own override, would it be possible to calculate that value from some other attribute for all hosts instead of having 1800 overrides, or maybe use a custom fact on the hosts to set the value?
Ok, I tested again and changing the file “_value.html.erb” makes loading the edit page ~10s faster. Unfortunately this doesnt make a big difference.
Can we provide further data/logs which can help to locate the problem? Or can we assist in any other way?
Some years ago we migrated to Foreman to have a Webfrontend for our customers, to make it easier for them. Now its “hard” to tell them, that they have to use the cli. I think the acceptance will not be the highest.
Your’re right, loading one parameter is faster then loading the whole class. The time to load depends on the number of overrides. A single midrange parameter (~600 overrides, we have quite a lot of them) takes ~25s to load, the whole class n-times longer.
Unfortunately its not possible to calculate this override or to use facts :(. Yesterday we switched to another way to deploy this information (beside puppet/foreman) and deleted all overrides from Foreman. It gave a huge performance “boost”, but actually that’s not the way how we want to automate or deploy settings.
As i mentioned some posts above the biggest performance boost came through “deactivating” the activateTooltips function. As a quick workaround, perhaps it is possible the add a setting for big foreman setups to deactivate the tooltips?
That is quite an interesting find - as it indicates that it isn’t the number of tooltips that causing the issue (that change should have led to a 2X improvement if it was).
The activateTooltips function is composed of 3 parts (for different types of tooltips) - each of them starting with el.find here:
Could you try commenting out each one of them separately so we could identify if one of them is responsible for the slowness?
Below you can find the results of my test. It seems that el.find('*[title]') is the “peformance killer”. I repeated every test 3 times to get a baseline. Between every test, i cleaned up the public assets folder, temp folder and recompiled assets//webpack.
Great finding! one last thing, just so we know if it’s the tooltip initialization that is slow or finding the right elements, can you please replace that part with:
let test = el.find('*[title]').not('*[rel],.fa,.pficon');
Looks like we have been initializing tooltips (and popovers) in a wrong way all this time, I believe I found a much better way to do it. Please open an issue on projects.theforeman.org and I’ll open a PR for you to test.
Replacing “el.find(’*[title]’) …” through your “let test…” results in ~12s load time.
Sure, i can do that, but give me some minutes. I’ll copy the link to this topic as soon as it’s opened
. The Puppetclass is assigned to ~ 1800 hosts and has 27 Smart Class Parameters. Thereof approx. the half of the parameters are enabled for override. I think the total count of matchers results in 1000-1500 overrides.
