There is only one Parameter with ~1800 overrides. In my opinion this is a range of overrides, which should be expected in some bigger environments (if you use Foreman as ENC).
And with Foreman 2.0 everything was ok. It wasnt the fastest, but “only” took ~8-12s to load this modul.
So there really was no changes which could result in this behaviour?
I dont want to appear impolite, but i need to ask. Do you recommend us to stay with foreman as enc in this scale (number of host, puppetclasses, overrides)?
You need to restart the server for the change to take affect.
As I mentioned, this is quite odd, as this page had no changes between 2.0 and 2.2 - so it’s quite difficult to understand why all of a sudden it became so slow
I believe the ENC functionality should still be fine, it’s only this page load that is problematic. An alternative option is to edit the parameter overrides using the API or CLI. Also, you can edit the specific parameters directly from Configure -> Smart Class Parameters menu, I would expect that all but the one that has 1852 overrides would load quite quickly (I tested locally with 150 overrides which seems to load with no issue). How long does it take to load that one parameter directly? Also, perhaps if practically every host has its own override, would it be possible to calculate that value from some other attribute for all hosts instead of having 1800 overrides, or maybe use a custom fact on the hosts to set the value?
Ok, I tested again and changing the file “_value.html.erb” makes loading the edit page ~10s faster. Unfortunately this doesnt make a big difference.
Can we provide further data/logs which can help to locate the problem? Or can we assist in any other way?
Some years ago we migrated to Foreman to have a Webfrontend for our customers, to make it easier for them. Now its “hard” to tell them, that they have to use the cli. I think the acceptance will not be the highest.
Your’re right, loading one parameter is faster then loading the whole class. The time to load depends on the number of overrides. A single midrange parameter (~600 overrides, we have quite a lot of them) takes ~25s to load, the whole class n-times longer.
Unfortunately its not possible to calculate this override or to use facts :(. Yesterday we switched to another way to deploy this information (beside puppet/foreman) and deleted all overrides from Foreman. It gave a huge performance “boost”, but actually that’s not the way how we want to automate or deploy settings.
As i mentioned some posts above the biggest performance boost came through “deactivating” the activateTooltips function. As a quick workaround, perhaps it is possible the add a setting for big foreman setups to deactivate the tooltips?
That is quite an interesting find - as it indicates that it isn’t the number of tooltips that causing the issue (that change should have led to a 2X improvement if it was).
The activateTooltips function is composed of 3 parts (for different types of tooltips) - each of them starting with el.find here:
Could you try commenting out each one of them separately so we could identify if one of them is responsible for the slowness?
Below you can find the results of my test. It seems that el.find('*[title]') is the “peformance killer”. I repeated every test 3 times to get a baseline. Between every test, i cleaned up the public assets folder, temp folder and recompiled assets//webpack.
Great finding! one last thing, just so we know if it’s the tooltip initialization that is slow or finding the right elements, can you please replace that part with:
let test = el.find('*[title]').not('*[rel],.fa,.pficon');
Looks like we have been initializing tooltips (and popovers) in a wrong way all this time, I believe I found a much better way to do it. Please open an issue on projects.theforeman.org and I’ll open a PR for you to test.
Replacing “el.find(’*[title]’) …” through your “let test…” results in ~12s load time.
Sure, i can do that, but give me some minutes. I’ll copy the link to this topic as soon as it’s opened
Sure, I tested again aaannnd the fix seems to work Now it takes 6-8s to load the page (previously without the fix 50+s). Thats a great performance boost. Thank you.
But one more thing, your patch is for the latest code of foreman (i think v2.3.X/2.4.X), right? I tested with 2.2.2 and did not found some of your diff’s in our copy of the source code, e.g. webpack/assets/javascripts/foreman_tools.js -> L137:139
The patch is against the current development branch. Looking at the 2.2 branch, this is before highlightTabErrors was moved to webpack, so the only change you need in application.js is changing:
just some of the numbers from this post so it’s easy for the reviewer to understand the impact - number of overrides on the page and time before/after would be great.
Also if you can confirm that the tooltips and popovers still display properly it would be great i tested it on my machine but always best to get another confirmation.
@tbrisker , this morning i started to rebuild our foreman package with your fix and deployed it on our integration system.
In this environment we recognized that the edit page of hosts now throw the error “Oops, we’re sorry but something went wrong comparison of Array with Array failed”.
I’m not yet sure, whether this error comes from the fix or from the build dependencies
This could not be caused by the fix since it only touches JS code for displaying tooltips, and that error message is coming from the server. there should be a full stacktrace in production.log (though you might need to change your logging level to debug to see it).
For some reason the classification is failing to properly sort the overrides for the host. is this on every host or just a specific one? could it be related to some change you made to the overrides in one of the previous attempts to speed up the puppet class forms? is the ENC output for the host correct or is it also erroring?
: 