Using tmpfs significantly reduces testing time

lzap · August 18, 2016, 7:41am

Hello,

this morning I observed that putting foreman git repo onto tmpfs folder
(e.g. /tmp/foreman) greatly reduces testing time by more than 50 % !

Running it from my checkout folder (Samsung SSD - 2014 edition):

real 27m24.425s
user 12m25.875s
sys 0m27.050s

And now from tmpfs:

real 12m21.645s
user 9m59.112s
sys 0m9.877s

I don't know if it's just the db/ directory or we do some temp files
during tests, but before I investigate this further, I'd like someone to
try this out and compare the numbers. I could made a mistake.

If I had to guess, I think sqlite does some filesystem sync operations
which are just noops on tmpfs (it's memory or swap right).

This would be great benefit on our Jenkins assuming there is tmpfs
available. I guess so (RHEL6+ has it). We do start always from scratch
there, so mounting/symlinking db/ should be relatively easy to do. And I
currently have about 35 MB in db/ folder, nothing that should kill the
runner.

···

-- Later, Lukas #lzap Zapletal

lzap · August 18, 2016, 8:22am

> If I had to guess, I think sqlite does some filesystem sync operations
> which are just noops on tmpfs (it's memory or swap right).

I can confirm that just copying db/test.sqlite3 onto tmpfs volume and
symlinking improves the time by 50 %.

We do test all our db platforms on Jenkins. Times are approx:

SQLite: 20 min
MySQL: 25 min
PostgreSQL: 35 min

Adding this trick won't necessarily shorten wait time, but it will save
some time that can be used for other tests.

It is worth looking on PostgreSQL/MySQL settings if there is any chance
of setting some faster IO mode as well.

···

-- Later, Lukas #lzap Zapletal

tbrisker · August 24, 2016, 10:38am

+1 to looking into implementing on Jenkins.
I would suggest even considering moving the entire /var/lib dir to tmpfs
instead of on disk so the other DBs will get a boost - assuming the
/var/lib dir is created anyways only for the worker's lifetime and
destroyed afterwards.
If this even gives us a 10% boost to test speed that would be a significant
step to reducing the Jenkins load.

···

On Thu, Aug 18, 2016 at 10:41 AM, Lukas Zapletal wrote:

Hello,

this morning I observed that putting foreman git repo onto tmpfs folder
(e.g. /tmp/foreman) greatly reduces testing time by more than 50 % !

Running it from my checkout folder (Samsung SSD - 2014 edition):

real 27m24.425s
user 12m25.875s
sys 0m27.050s

And now from tmpfs:

real 12m21.645s
user 9m59.112s
sys 0m9.877s

I don’t know if it’s just the db/ directory or we do some temp files
during tests, but before I investigate this further, I’d like someone to
try this out and compare the numbers. I could made a mistake.

If I had to guess, I think sqlite does some filesystem sync operations
which are just noops on tmpfs (it’s memory or swap right).

This would be great benefit on our Jenkins assuming there is tmpfs
available. I guess so (RHEL6+ has it). We do start always from scratch
there, so mounting/symlinking db/ should be relatively easy to do. And I
currently have about 35 MB in db/ folder, nothing that should kill the
runner.

–
Later,
Lukas #lzap Zapletal

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Have a nice day,
Tomer Brisker
Red Hat Engineering

lzap · August 19, 2016, 6:57am

> I can confirm that just copying db/test.sqlite3 onto tmpfs volume and
> symlinking improves the time by 50 %.

Just for those who want to test it out, I made a script I am going to
use for local testing. Saving 10 minutes is absolutely worth it:

github.com

lzap/bin-public/blob/master/bet

#/bin/bash
cleanup() {
  test -h db/test.sqlite3 && mv -f /tmp/test.sqlite3 db/
}
test -f db/test.sqlite3 && mv db/test.sqlite3 /tmp && ln -s /tmp/test.sqlite3 db/test.sqlite3
trap "cleanup" SIGHUP SIGINT SIGTERM
FOREMAN_NO_DEPRECATIONS=1 DISABLE_SPRING=true bundle exec rake test $* -v
cleanup

LZ

···

-- Later, Lukas #lzap Zapletal

lzap · August 24, 2016, 1:36pm

> I would suggest even considering moving the entire /var/lib dir to tmpfs
> instead of on disk so the other DBs will get a boost - assuming the
> /var/lib dir is created anyways only for the worker's lifetime and
> destroyed afterwards.

That's perhaps too much. But fine tuning PostgreSQL and MySQL instances
could help a lot (giving up some atomicity or consistency in the storage
subsystem).

···

-- Later, Lukas #lzap Zapletal

Shimon_Shtein · August 21, 2016, 6:35am

I heard that there is an option to use sqlite in-memory.
I gave to admit I didn't try it, but I think it worth mentioning.

···

On Friday, August 19, 2016 at 9:57:38 AM UTC+3, Lukas Zapletal wrote: > > > I can confirm that just copying db/test.sqlite3 onto tmpfs volume and > > symlinking improves the time by 50 %. > > Just for those who want to test it out, I made a script I am going to > use for local testing. Saving 10 minutes is absolutely worth it: > > https://github.com/lzap/bin-public/blob/master/bet > > LZ > > -- > Later, > Lukas #lzap Zapletal >

ohadlevy · August 21, 2016, 11:37am

>
> I heard that there is an option to use sqlite in-memory.
> I gave to admit I didn't try it, but I think it worth mentioning.
>

first google hit:
http://www.calicowebdev.com/2011/01/25/rails-3-sqlite-3-in-memory-databases/

Ohad

···

On Sun, Aug 21, 2016 at 9:35 AM, wrote:

On Friday, August 19, 2016 at 9:57:38 AM UTC+3, Lukas Zapletal wrote:

I can confirm that just copying db/test.sqlite3 onto tmpfs volume and
symlinking improves the time by 50 %.

Just for those who want to test it out, I made a script I am going to
use for local testing. Saving 10 minutes is absolutely worth it:

https://github.com/lzap/bin-public/blob/master/bet

LZ

–
Later,
Lukas #lzap Zapletal

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

lzap · August 22, 2016, 11:44am

> > I heard that there is an option to use sqlite in-memory.
> > I gave to admit I didn't try it, but I think it worth mentioning.
>
> first google hit:
> http://www.calicowebdev.com/2011/01/25/rails-3-sqlite-3-in-memory-databases/

Those first hits I have considered that of course, it does not work.
The moment connection is closed the database dies. Migration, db load
and test rake tasks are executed as separate processes.

···

-- Later, Lukas #lzap Zapletal

Shimon_Shtein · August 22, 2016, 1:39pm

I have manged to hack the memory DB to work [1], but for me, the time
difference wasn't that big (~10%).

It would be nice to see the differences.

[1]
https://github.com/ShimShtein/foreman/compare/15846...ShimShtein:memory_try?expand=1

···

On Monday, August 22, 2016 at 2:44:07 PM UTC+3, Lukas Zapletal wrote: > > > > I heard that there is an option to use sqlite in-memory. > > > I gave to admit I didn't try it, but I think it worth mentioning. > > > > first google hit: > > > http://www.calicowebdev.com/2011/01/25/rails-3-sqlite-3-in-memory-databases/ > > Those first hits :-) I have considered that of course, it does not work. > The moment connection is closed the database dies. Migration, db load > and test rake tasks are executed as separate processes. > > -- > Later, > Lukas #lzap Zapletal >

lzap · August 24, 2016, 9:10am

> It would be nice to see the differences.

It's almost the same speedup as with symlink (the difference is loading
the database three times perhaps):

real 13m32.018s
user 11m13.229s
sys 0m12.004s

What is your file system? That might do the difference.

···

-- Later, Lukas #lzap Zapletal

Shimon_Shtein · August 24, 2016, 9:35am

I have ext4 for my home and tmpfs for /tmp.
I haven't tried your solution (symlink), but if you say it's pretty much
the same as in-memory, I will surely switch to it.

···

On Wednesday, August 24, 2016 at 12:10:05 PM UTC+3, Lukas Zapletal wrote: > > > It would be nice to see the differences. > > It's almost the same speedup as with symlink (the difference is loading > the database three times perhaps): > > real 13m32.018s > user 11m13.229s > sys 0m12.004s > > What is your file system? That might do the difference. > > -- > Later, > Lukas #lzap Zapletal >

lzap · August 24, 2016, 1:34pm

Please file a PR so we can compare speedup on Jenkins and discuss
details.

LZ

···

On Wed, Aug 24, 2016 at 02:35:21AM -0700, Shimon Shtein wrote: > > I have ext4 for my home and tmpfs for /tmp. > I haven't tried your solution (symlink), but if you say it's pretty much > the same as in-memory, I will surely switch to it. > > > On Wednesday, August 24, 2016 at 12:10:05 PM UTC+3, Lukas Zapletal wrote: > > > > > It would be nice to see the differences. > > > > It's almost the same speedup as with symlink (the difference is loading > > the database three times perhaps): > > > > real 13m32.018s > > user 11m13.229s > > sys 0m12.004s > > > > What is your file system? That might do the difference. > > > > -- > > Later, > > Lukas #lzap Zapletal > > > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas #lzap Zapletal