September 21, 2021, 9:26am
it looks like
npm install takes around 21 minutes on our CI, webpack compilation takes 5 minutes. Is there a problem with Node for plugin tests? That does not look healthy.
For this reason, discovery plugin tests are failing reaching 2 hours. Total time of Ruby tests is one hour so hardware/VM seems reasonable sized (its like 40 minutes on my machine).
Can someone help to investigate why JS stack for plugins is so slow? Perhaps its not using the correct node driver?
If we don’t find any solution, can we make the 2 hours limit a bit longer?
I can’t do anything about the limit, but I think we could at least cache our npm dependencies as GH actions are doing for us. GH has it’s own npm registry, that caches all the requests to npm, we could do the same for ours.
We could just limit our dependency tree as it is huge and I’d argue it is not necessary. It is very hard work obviously, but it will need to be done at some point for sure.
Tests that runs for 2 hours are almost useless.
September 21, 2021, 1:33pm
I remember someone from our
@infra team saying we already cache something on IRC. Guys discussed that Node might be creating ton of small files and our instances use spinning drives, that could be a bottleneck too.
This needs to be investigated, maybe it is just misconfiguration. Or maybe we can tune the OS for better performance.
September 29, 2021, 10:45am
Bump, guys its failing most of the time and I need to get this resolved.
I propose to increase jenkins limit to 2 hours 30 minutes until we figure this out. How do I do this?
This has been bugging me for over a year now:
I am out of ideas so seeking for help. Discovery PR test on jenkins calls the standard plugin test rake task which is supposed to test plugin and then call also all core tests with the plugin installed to find it if the plugin does not break something.
Recently, tests started to hit 2 hours limit we have on jenkins, for some reason only for Ruby 2.7 (which is supposed to be the fastest out of the three). I can’t find why this happens.
Normally, core tests run for about 40 minutes. But di…
November 5, 2021, 7:02pm
Have you noticed problems/improvements since this:
I just installed
node01.jenkins.conova.theforeman.org and removed the fast label from our OSUOSL builders.
We’ve observed that Foreman CI tests took about 30 minutes on our new Conova node, which is the same as on our AWS nodes. However, the OSUOSL nodes took about 60 to 90 minutes. Removing the fast label means jobs should be scheduled on the actually fast machines. We’ll have to see if this brings capacity issues. If needed, we can install another node on the conova machine.
Note that was done about a month ago.
November 8, 2021, 10:27am