I'm running the whole puppet/forman stack on a single vm. (debian wheezy).
This vm does not run any other services. Theforeman setup has not been
altered.
We have about 400 nodes, They're never on-line all together.
We're running into to some issues with the setup. Postgresql is constantly
trashing our disks. While writing this, it has already accumulated over 8gb
of read data and written about 200mb (iotop -oa).
The postgresql instance is dedicated to foreman. The cpu load is 15%
We can't find any reason for this. It's been going on for about 2monts
now(and maybe longer, but didn't have monitoring on the box before)
Can anyone suggest where we should start looking? This behavior is causing
quite a performance penalty on our other vm's… (15Gb data read by now
)
> Can anyone suggest where we should start looking? This behavior is causing
> quite a performance penalty on our other vm's… (15Gb data read by now
Hello,
I'd start with setting log_statement to "all" for small period of time
(watching pg_logs with tail) to collect some SQL queries are around. I
think pgsql does not log how long it took to execute queries, but you
can pick those which look suspicious and execute (SELECTs I mean) them
in a console to measure the performance.
Also try changing the IO scheduler to noop or deadline, this helps a lot
for database performance.
Kind regards,
Jorick
···
On 11/24/2014 02:17 PM, Bart bartbart wrote:
> Hi All,
>
> I'm running the whole puppet/forman stack on a single vm. (debian
> wheezy). This vm does not run any other services. Theforeman setup has
> not been altered.
> We have about 400 nodes, They're never on-line all together.
>
> We're running into to some issues with the setup. Postgresql is
> constantly trashing our disks. While writing this, it has already
> accumulated over 8gb of read data and written about 200mb (iotop -oa).
> The postgresql instance is dedicated to foreman. The cpu load is 15%
>
> We can't find any reason for this. It's been going on for about 2monts
> now(and maybe longer, but didn't have monitoring on the box before)
> Can anyone suggest where we should start looking? This behavior is
> causing quite a performance penalty on our other vm's... (15Gb data
> read by now :( )
>
> So far, we've only tried:
>
> * restarting the box
> * Increase postgresql shared buffers
>
> The database seems to be arround 16gb in size.
>
>
>
>
> Regards,
>
> Bart
And for the database size, check where all the space went:
psql foreman
SELECT schemaname,relname,n_live_tup
FROM pg_stat_user_tables
ORDER BY n_live_tup DESC;
SELECT relname AS "relation", pg_size_pretty(pg_relation_size(C.oid)) AS
"size"
FROM pg_class C LEFT JOIN pg_namespace N ON (N.oid = C.relnamespace)
WHERE nspname NOT IN ('pg_catalog', 'information_schema')
ORDER BY pg_relation_size(C.oid) DESC
LIMIT 10;
Kind regards,
Jorick
Met vriendelijke groet, With kind regards,
Jorick Astrego
Netbulae Virtualization Experts
···
On 11/24/2014 02:17 PM, Bart bartbart wrote:
> Hi All,
>
> I'm running the whole puppet/forman stack on a single vm. (debian
> wheezy). This vm does not run any other services. Theforeman setup has
> not been altered.
> We have about 400 nodes, They're never on-line all together.
>
> We're running into to some issues with the setup. Postgresql is
> constantly trashing our disks. While writing this, it has already
> accumulated over 8gb of read data and written about 200mb (iotop -oa).
> The postgresql instance is dedicated to foreman. The cpu load is 15%
>
> We can't find any reason for this. It's been going on for about 2monts
> now(and maybe longer, but didn't have monitoring on the box before)
> Can anyone suggest where we should start looking? This behavior is
> causing quite a performance penalty on our other vm's... (15Gb data
> read by now :( )
>
> So far, we've only tried:
>
> * restarting the box
> * Increase postgresql shared buffers
>
> The database seems to be arround 16gb in size.
>
>
>
>
> Regards,
>
> Bart
