Trends + High Load

Hello Foreman users,

I have a problem with a very high load on Foreman server (around 30.0).
There are numerous processess with trends:counter during this high load.
I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all the
processess with trends:counter, but they keep on popping up.

I've got one trend: Memory-free-MB. The GUI for the trends works awfully
slow, and when I click the delete option for this trend I get timeouted
after ~20 minutes. I've also tried the command-line (foreman-rake
trends:clean) but that does not help. The problem can be related to
postgresql server, because the trends:counter process keeps querying DB.
I've tried to vacuum PSQL but it didn't improve anything at all.

Is there any way I can remove the trends manually (either from DB, or by
URL, or command-line)? Or maybe there is a way to optimize postgresql
database so the trends would work properly? Or my high load is related to
some other thing (so how to check it). I would appreciate any help, and if
you need more information about my environment please post.

Thanks in advance,
Erick

Hi Erick,

  1. What version of Foreman are you running?
  2. Can you please check the row count for trend_counters table ('SELECT
    count(*) FROM trend_counters")?

If my hunch is right, you will see very large row count there. If this is
the case I have two things for you.
First: I did some changes to the way the data is stored in this table, so
in 1.9 beta it should be more compact.
Second, The table stores data points from the beginning of the time, so you
can delete old records. ("DELETE FROM trend_counters WHERE created_at <
…")

Don't forget to back up your data before those database manipulations!!!

Shimon

··· On Tuesday, July 7, 2015 at 1:24:44 PM UTC+3, Ziwi wrote: > > Hello Foreman users, > > I have a problem with a very high load on Foreman server (around 30.0). > There are numerous processess with trends:counter during this high load. > I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all the > processess with trends:counter, but they keep on popping up. > > I've got one trend: Memory-free-MB. The GUI for the trends works awfully > slow, and when I click the delete option for this trend I get timeouted > after ~20 minutes. I've also tried the command-line (foreman-rake > trends:clean) but that does not help. The problem can be related to > postgresql server, because the trends:counter process keeps querying DB. > I've tried to vacuum PSQL but it didn't improve anything at all. > > Is there any way I can remove the trends manually (either from DB, or by > URL, or command-line)? Or maybe there is a way to optimize postgresql > database so the trends would work properly? Or my high load is related to > some other thing (so how to check it). I would appreciate any help, and if > you need more information about my environment please post. > > Thanks in advance, > Erick >

> Hello Foreman users,
>
> I have a problem with a very high load on Foreman server (around 30.0).
> There are numerous processess with trends:counter during this high load.
> I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all the
> processess with trends:counter, but they keep on popping up.

Trends are set as a cron job, run crontab -e and delete it if you want
to, but obviously you'll not get data anymore if you use the feature. As
Shimon mentioned, on the newer Foreman versions (1.8 and 1.9) he made some
performance improvements that hopefully alleviate your problem.

··· On 07/07, Ziwi wrote: > > I've got one trend: Memory-free-MB. The GUI for the trends works awfully > slow, and when I click the delete option for this trend I get timeouted > after ~20 minutes. I've also tried the command-line (foreman-rake > trends:clean) but that does not help. The problem can be related to > postgresql server, because the trends:counter process keeps querying DB. > I've tried to vacuum PSQL but it didn't improve anything at all. > > Is there any way I can remove the trends manually (either from DB, or by > URL, or command-line)? Or maybe there is a way to optimize postgresql > database so the trends would work properly? Or my high load is related to > some other thing (so how to check it). I would appreciate any help, and if > you need more information about my environment please post. > > Thanks in advance, > Erick > > -- > You received this message because you are subscribed to the Google Groups "Foreman users" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-users+unsubscribe@googlegroups.com. > To post to this group, send email to foreman-users@googlegroups.com. > Visit this group at http://groups.google.com/group/foreman-users. > For more options, visit https://groups.google.com/d/optout.


Daniel Lobato Garcia

@eLobatoss
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

Hello Shimon,

  1. I am using Foreman 1.8.5, updated two days ago from 1.8.2
  2. The row count is 95087572, so massive

I'm glad that you will be optimizing the trends in 1.9, however I'd rather
not update my Foreman instance to the beta version since it's a production
environment. So that leaves me with clearing old trends from db, just like
you said.

Thanks for your answers guys, I'll try this and see if the high load
problem will appear again.

Best regards,
Erick

W dniu wtorek, 7 lipca 2015 14:45:19 UTC+2 użytkownik ssh...@redhat.com
napisał:

··· > > Hi Erick, > > 1. What version of Foreman are you running? > 2. Can you please check the row count for trend_counters table ('SELECT > count(*) FROM trend_counters")? > > If my hunch is right, you will see very large row count there. If this is > the case I have two things for you. > First: I did some changes to the way the data is stored in this table, so > in 1.9 beta it should be more compact. > Second, The table stores data points from the beginning of the time, so > you can delete old records. ("DELETE FROM trend_counters WHERE created_at < > ...") > > Don't forget to back up your data before those database manipulations!!! > > Shimon > > On Tuesday, July 7, 2015 at 1:24:44 PM UTC+3, Ziwi wrote: >> >> Hello Foreman users, >> >> I have a problem with a very high load on Foreman server (around 30.0). >> There are numerous processess with trends:counter during this high load. >> I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all the >> processess with trends:counter, but they keep on popping up. >> >> I've got one trend: Memory-free-MB. The GUI for the trends works awfully >> slow, and when I click the delete option for this trend I get timeouted >> after ~20 minutes. I've also tried the command-line (foreman-rake >> trends:clean) but that does not help. The problem can be related to >> postgresql server, because the trends:counter process keeps querying DB. >> I've tried to vacuum PSQL but it didn't improve anything at all. >> >> Is there any way I can remove the trends manually (either from DB, or by >> URL, or command-line)? Or maybe there is a way to optimize postgresql >> database so the trends would work properly? Or my high load is related to >> some other thing (so how to check it). I would appreciate any help, and if >> you need more information about my environment please post. >> >> Thanks in advance, >> Erick >> >

> > Hello Foreman users,
> >
> > I have a problem with a very high load on Foreman server (around 30.0).
> > There are numerous processess with trends:counter during this high load.
> > I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all the
> > processess with trends:counter, but they keep on popping up.
>
> Trends are set as a cron job, run crontab -e and delete it if you want
> to, but obviously you'll not get data anymore if you use the feature. As
> Shimon mentioned, on the newer Foreman versions (1.8 and 1.9) he made some
> performance improvements that hopefully alleviate your problem.
>
just to mention, changes in 1.8 are minor improvements, 1.9 is a huge
improvement in trends performance.

Ohad

··· On Tue, Jul 7, 2015 at 3:50 PM, Daniel Lobato Garcia wrote: > On 07/07, Ziwi wrote:

I’ve got one trend: Memory-free-MB. The GUI for the trends works awfully
slow, and when I click the delete option for this trend I get timeouted
after ~20 minutes. I’ve also tried the command-line (foreman-rake
trends:clean) but that does not help. The problem can be related to
postgresql server, because the trends:counter process keeps querying DB.
I’ve tried to vacuum PSQL but it didn’t improve anything at all.

Is there any way I can remove the trends manually (either from DB, or by
URL, or command-line)? Or maybe there is a way to optimize postgresql
database so the trends would work properly? Or my high load is related to
some other thing (so how to check it). I would appreciate any help, and
if
you need more information about my environment please post.

Thanks in advance,
Erick


You received this message because you are subscribed to the Google
Groups “Foreman users” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.


Daniel Lobato Garcia

@eLobatoss
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-users+unsubscribe@googlegroups.com.
To post to this group, send email to foreman-users@googlegroups.com.
Visit this group at http://groups.google.com/group/foreman-users.
For more options, visit https://groups.google.com/d/optout.

Hi Eric,

Can you upload somewhere public the most recent 500K records?

It would help me a lot with testing a real world data to see how much do I
improve the situation.

Thanks,
Shimon.

··· On Tuesday, July 7, 2015 at 4:23:59 PM UTC+3, Ziwi wrote: > > Hello Shimon, > > 1. I am using Foreman 1.8.5, updated two days ago from 1.8.2 > 2. The row count is 95087572, so massive > > I'm glad that you will be optimizing the trends in 1.9, however I'd rather > not update my Foreman instance to the beta version since it's a production > environment. So that leaves me with clearing old trends from db, just like > you said. > > Thanks for your answers guys, I'll try this and see if the high load > problem will appear again. > > Best regards, > Erick > > W dniu wtorek, 7 lipca 2015 14:45:19 UTC+2 użytkownik ssh...@redhat.com > napisał: >> >> Hi Erick, >> >> 1. What version of Foreman are you running? >> 2. Can you please check the row count for trend_counters table ('SELECT >> count(*) FROM trend_counters")? >> >> If my hunch is right, you will see very large row count there. If this is >> the case I have two things for you. >> First: I did some changes to the way the data is stored in this table, so >> in 1.9 beta it should be more compact. >> Second, The table stores data points from the beginning of the time, so >> you can delete old records. ("DELETE FROM trend_counters WHERE created_at < >> ...") >> >> Don't forget to back up your data before those database manipulations!!! >> >> Shimon >> >> On Tuesday, July 7, 2015 at 1:24:44 PM UTC+3, Ziwi wrote: >>> >>> Hello Foreman users, >>> >>> I have a problem with a very high load on Foreman server (around 30.0). >>> There are numerous processess with trends:counter during this high load. >>> I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all the >>> processess with trends:counter, but they keep on popping up. >>> >>> I've got one trend: Memory-free-MB. The GUI for the trends works awfully >>> slow, and when I click the delete option for this trend I get timeouted >>> after ~20 minutes. I've also tried the command-line (foreman-rake >>> trends:clean) but that does not help. The problem can be related to >>> postgresql server, because the trends:counter process keeps querying DB. >>> I've tried to vacuum PSQL but it didn't improve anything at all. >>> >>> Is there any way I can remove the trends manually (either from DB, or by >>> URL, or command-line)? Or maybe there is a way to optimize postgresql >>> database so the trends would work properly? Or my high load is related to >>> some other thing (so how to check it). I would appreciate any help, and if >>> you need more information about my environment please post. >>> >>> Thanks in advance, >>> Erick >>> >>

Simon,

Unfortunately I've already issued DELETE query for rows before 1st June.
Foreman is working in production since March, so I will see how many rows
will remain and I if it will be a suitable amount I'll upload them for you.

Best regards,
Eric

W dniu wtorek, 7 lipca 2015 15:39:37 UTC+2 użytkownik ssh...@redhat.com
napisał:

··· > > Hi Eric, > > Can you upload somewhere public the most recent 500K records? > > It would help me a lot with testing a real world data to see how much do I > improve the situation. > > Thanks, > Shimon. > > > > On Tuesday, July 7, 2015 at 4:23:59 PM UTC+3, Ziwi wrote: >> >> Hello Shimon, >> >> 1. I am using Foreman 1.8.5, updated two days ago from 1.8.2 >> 2. The row count is 95087572, so massive >> >> I'm glad that you will be optimizing the trends in 1.9, however I'd >> rather not update my Foreman instance to the beta version since it's a >> production environment. So that leaves me with clearing old trends from db, >> just like you said. >> >> Thanks for your answers guys, I'll try this and see if the high load >> problem will appear again. >> >> Best regards, >> Erick >> >> W dniu wtorek, 7 lipca 2015 14:45:19 UTC+2 użytkownik ssh...@redhat.com >> napisał: >>> >>> Hi Erick, >>> >>> 1. What version of Foreman are you running? >>> 2. Can you please check the row count for trend_counters table ('SELECT >>> count(*) FROM trend_counters")? >>> >>> If my hunch is right, you will see very large row count there. If this >>> is the case I have two things for you. >>> First: I did some changes to the way the data is stored in this table, >>> so in 1.9 beta it should be more compact. >>> Second, The table stores data points from the beginning of the time, so >>> you can delete old records. ("DELETE FROM trend_counters WHERE created_at < >>> ...") >>> >>> Don't forget to back up your data before those database manipulations!!! >>> >>> Shimon >>> >>> On Tuesday, July 7, 2015 at 1:24:44 PM UTC+3, Ziwi wrote: >>>> >>>> Hello Foreman users, >>>> >>>> I have a problem with a very high load on Foreman server (around 30.0). >>>> There are numerous processess with trends:counter during this high load. >>>> I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all the >>>> processess with trends:counter, but they keep on popping up. >>>> >>>> I've got one trend: Memory-free-MB. The GUI for the trends works >>>> awfully slow, and when I click the delete option for this trend I get >>>> timeouted after ~20 minutes. I've also tried the command-line (foreman-rake >>>> trends:clean) but that does not help. The problem can be related to >>>> postgresql server, because the trends:counter process keeps querying DB. >>>> I've tried to vacuum PSQL but it didn't improve anything at all. >>>> >>>> Is there any way I can remove the trends manually (either from DB, or >>>> by URL, or command-line)? Or maybe there is a way to optimize postgresql >>>> database so the trends would work properly? Or my high load is related to >>>> some other thing (so how to check it). I would appreciate any help, and if >>>> you need more information about my environment please post. >>>> >>>> Thanks in advance, >>>> Erick >>>> >>>

Thanks Eric, it will be fantastic!

··· On Tuesday, July 7, 2015 at 4:58:45 PM UTC+3, Ziwi wrote: > > Simon, > > Unfortunately I've already issued DELETE query for rows before 1st June. > Foreman is working in production since March, so I will see how many rows > will remain and I if it will be a suitable amount I'll upload them for you. > > Best regards, > Eric > > W dniu wtorek, 7 lipca 2015 15:39:37 UTC+2 użytkownik ssh...@redhat.com > napisał: >> >> Hi Eric, >> >> Can you upload somewhere public the most recent 500K records? >> >> It would help me a lot with testing a real world data to see how much do >> I improve the situation. >> >> Thanks, >> Shimon. >> >> >> >> On Tuesday, July 7, 2015 at 4:23:59 PM UTC+3, Ziwi wrote: >>> >>> Hello Shimon, >>> >>> 1. I am using Foreman 1.8.5, updated two days ago from 1.8.2 >>> 2. The row count is 95087572, so massive >>> >>> I'm glad that you will be optimizing the trends in 1.9, however I'd >>> rather not update my Foreman instance to the beta version since it's a >>> production environment. So that leaves me with clearing old trends from db, >>> just like you said. >>> >>> Thanks for your answers guys, I'll try this and see if the high load >>> problem will appear again. >>> >>> Best regards, >>> Erick >>> >>> W dniu wtorek, 7 lipca 2015 14:45:19 UTC+2 użytkownik ssh...@redhat.com >>> napisał: >>>> >>>> Hi Erick, >>>> >>>> 1. What version of Foreman are you running? >>>> 2. Can you please check the row count for trend_counters table ('SELECT >>>> count(*) FROM trend_counters")? >>>> >>>> If my hunch is right, you will see very large row count there. If this >>>> is the case I have two things for you. >>>> First: I did some changes to the way the data is stored in this table, >>>> so in 1.9 beta it should be more compact. >>>> Second, The table stores data points from the beginning of the time, so >>>> you can delete old records. ("DELETE FROM trend_counters WHERE created_at < >>>> ...") >>>> >>>> Don't forget to back up your data before those database manipulations!!! >>>> >>>> Shimon >>>> >>>> On Tuesday, July 7, 2015 at 1:24:44 PM UTC+3, Ziwi wrote: >>>>> >>>>> Hello Foreman users, >>>>> >>>>> I have a problem with a very high load on Foreman server (around >>>>> 30.0). There are numerous processess with trends:counter during this high >>>>> load. I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all >>>>> the processess with trends:counter, but they keep on popping up. >>>>> >>>>> I've got one trend: Memory-free-MB. The GUI for the trends works >>>>> awfully slow, and when I click the delete option for this trend I get >>>>> timeouted after ~20 minutes. I've also tried the command-line (foreman-rake >>>>> trends:clean) but that does not help. The problem can be related to >>>>> postgresql server, because the trends:counter process keeps querying DB. >>>>> I've tried to vacuum PSQL but it didn't improve anything at all. >>>>> >>>>> Is there any way I can remove the trends manually (either from DB, or >>>>> by URL, or command-line)? Or maybe there is a way to optimize postgresql >>>>> database so the trends would work properly? Or my high load is related to >>>>> some other thing (so how to check it). I would appreciate any help, and if >>>>> you need more information about my environment please post. >>>>> >>>>> Thanks in advance, >>>>> Erick >>>>> >>>>

Hi Simon,

I've attached the recent 500k records as you requested.

FYI, my delete query erased 50M records, leaving 45M, but the problem with
high load still exists. I'll try to delete the rest of records, then turn
off trends just to be sure this is the cause.

Best regards,
Eric

W dniu wtorek, 7 lipca 2015 16:08:03 UTC+2 użytkownik ssh...@redhat.com
napisał:

output2.csv.gz (7.09 KB)

··· > > Thanks Eric, it will be fantastic! > > On Tuesday, July 7, 2015 at 4:58:45 PM UTC+3, Ziwi wrote: >> >> Simon, >> >> Unfortunately I've already issued DELETE query for rows before 1st June. >> Foreman is working in production since March, so I will see how many rows >> will remain and I if it will be a suitable amount I'll upload them for you. >> >> Best regards, >> Eric >> >> W dniu wtorek, 7 lipca 2015 15:39:37 UTC+2 użytkownik ssh...@redhat.com >> napisał: >>> >>> Hi Eric, >>> >>> Can you upload somewhere public the most recent 500K records? >>> >>> It would help me a lot with testing a real world data to see how much do >>> I improve the situation. >>> >>> Thanks, >>> Shimon. >>> >>> >>> >>> On Tuesday, July 7, 2015 at 4:23:59 PM UTC+3, Ziwi wrote: >>>> >>>> Hello Shimon, >>>> >>>> 1. I am using Foreman 1.8.5, updated two days ago from 1.8.2 >>>> 2. The row count is 95087572, so massive >>>> >>>> I'm glad that you will be optimizing the trends in 1.9, however I'd >>>> rather not update my Foreman instance to the beta version since it's a >>>> production environment. So that leaves me with clearing old trends from db, >>>> just like you said. >>>> >>>> Thanks for your answers guys, I'll try this and see if the high load >>>> problem will appear again. >>>> >>>> Best regards, >>>> Erick >>>> >>>> W dniu wtorek, 7 lipca 2015 14:45:19 UTC+2 użytkownik ssh...@redhat.com >>>> napisał: >>>>> >>>>> Hi Erick, >>>>> >>>>> 1. What version of Foreman are you running? >>>>> 2. Can you please check the row count for trend_counters table >>>>> ('SELECT count(*) FROM trend_counters")? >>>>> >>>>> If my hunch is right, you will see very large row count there. If this >>>>> is the case I have two things for you. >>>>> First: I did some changes to the way the data is stored in this table, >>>>> so in 1.9 beta it should be more compact. >>>>> Second, The table stores data points from the beginning of the time, >>>>> so you can delete old records. ("DELETE FROM trend_counters WHERE >>>>> created_at < ...") >>>>> >>>>> Don't forget to back up your data before those database >>>>> manipulations!!! >>>>> >>>>> Shimon >>>>> >>>>> On Tuesday, July 7, 2015 at 1:24:44 PM UTC+3, Ziwi wrote: >>>>>> >>>>>> Hello Foreman users, >>>>>> >>>>>> I have a problem with a very high load on Foreman server (around >>>>>> 30.0). There are numerous processess with trends:counter during this high >>>>>> load. I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all >>>>>> the processess with trends:counter, but they keep on popping up. >>>>>> >>>>>> I've got one trend: Memory-free-MB. The GUI for the trends works >>>>>> awfully slow, and when I click the delete option for this trend I get >>>>>> timeouted after ~20 minutes. I've also tried the command-line (foreman-rake >>>>>> trends:clean) but that does not help. The problem can be related to >>>>>> postgresql server, because the trends:counter process keeps querying DB. >>>>>> I've tried to vacuum PSQL but it didn't improve anything at all. >>>>>> >>>>>> Is there any way I can remove the trends manually (either from DB, or >>>>>> by URL, or command-line)? Or maybe there is a way to optimize postgresql >>>>>> database so the trends would work properly? Or my high load is related to >>>>>> some other thing (so how to check it). I would appreciate any help, and if >>>>>> you need more information about my environment please post. >>>>>> >>>>>> Thanks in advance, >>>>>> Erick >>>>>> >>>>>

Great! I got the file.

If it's not much trouble, can you export the trends table too? It will save
me a lot of work…

Thanks,
Shimon.

··· On Wednesday, July 8, 2015 at 10:54:35 AM UTC+3, Ziwi wrote: > > Hi Simon, > > I've attached the recent 500k records as you requested. > > FYI, my delete query erased 50M records, leaving 45M, but the problem with > high load still exists. I'll try to delete the rest of records, then turn > off trends just to be sure this is the cause. > > Best regards, > Eric > > W dniu wtorek, 7 lipca 2015 16:08:03 UTC+2 użytkownik ssh...@redhat.com > napisał: >> >> Thanks Eric, it will be fantastic! >> >> On Tuesday, July 7, 2015 at 4:58:45 PM UTC+3, Ziwi wrote: >>> >>> Simon, >>> >>> Unfortunately I've already issued DELETE query for rows before 1st June. >>> Foreman is working in production since March, so I will see how many rows >>> will remain and I if it will be a suitable amount I'll upload them for you. >>> >>> Best regards, >>> Eric >>> >>> W dniu wtorek, 7 lipca 2015 15:39:37 UTC+2 użytkownik ssh...@redhat.com >>> napisał: >>>> >>>> Hi Eric, >>>> >>>> Can you upload somewhere public the most recent 500K records? >>>> >>>> It would help me a lot with testing a real world data to see how much >>>> do I improve the situation. >>>> >>>> Thanks, >>>> Shimon. >>>> >>>> >>>> >>>> On Tuesday, July 7, 2015 at 4:23:59 PM UTC+3, Ziwi wrote: >>>>> >>>>> Hello Shimon, >>>>> >>>>> 1. I am using Foreman 1.8.5, updated two days ago from 1.8.2 >>>>> 2. The row count is 95087572, so massive >>>>> >>>>> I'm glad that you will be optimizing the trends in 1.9, however I'd >>>>> rather not update my Foreman instance to the beta version since it's a >>>>> production environment. So that leaves me with clearing old trends from db, >>>>> just like you said. >>>>> >>>>> Thanks for your answers guys, I'll try this and see if the high load >>>>> problem will appear again. >>>>> >>>>> Best regards, >>>>> Erick >>>>> >>>>> W dniu wtorek, 7 lipca 2015 14:45:19 UTC+2 użytkownik >>>>> ssh...@redhat.com napisał: >>>>>> >>>>>> Hi Erick, >>>>>> >>>>>> 1. What version of Foreman are you running? >>>>>> 2. Can you please check the row count for trend_counters table >>>>>> ('SELECT count(*) FROM trend_counters")? >>>>>> >>>>>> If my hunch is right, you will see very large row count there. If >>>>>> this is the case I have two things for you. >>>>>> First: I did some changes to the way the data is stored in this >>>>>> table, so in 1.9 beta it should be more compact. >>>>>> Second, The table stores data points from the beginning of the time, >>>>>> so you can delete old records. ("DELETE FROM trend_counters WHERE >>>>>> created_at < ...") >>>>>> >>>>>> Don't forget to back up your data before those database >>>>>> manipulations!!! >>>>>> >>>>>> Shimon >>>>>> >>>>>> On Tuesday, July 7, 2015 at 1:24:44 PM UTC+3, Ziwi wrote: >>>>>>> >>>>>>> Hello Foreman users, >>>>>>> >>>>>>> I have a problem with a very high load on Foreman server (around >>>>>>> 30.0). There are numerous processess with trends:counter during this high >>>>>>> load. I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all >>>>>>> the processess with trends:counter, but they keep on popping up. >>>>>>> >>>>>>> I've got one trend: Memory-free-MB. The GUI for the trends works >>>>>>> awfully slow, and when I click the delete option for this trend I get >>>>>>> timeouted after ~20 minutes. I've also tried the command-line (foreman-rake >>>>>>> trends:clean) but that does not help. The problem can be related to >>>>>>> postgresql server, because the trends:counter process keeps querying DB. >>>>>>> I've tried to vacuum PSQL but it didn't improve anything at all. >>>>>>> >>>>>>> Is there any way I can remove the trends manually (either from DB, >>>>>>> or by URL, or command-line)? Or maybe there is a way to optimize postgresql >>>>>>> database so the trends would work properly? Or my high load is related to >>>>>>> some other thing (so how to check it). I would appreciate any help, and if >>>>>>> you need more information about my environment please post. >>>>>>> >>>>>>> Thanks in advance, >>>>>>> Erick >>>>>>> >>>>>>

Simon, I'm sorry - I've updated Foreman from my 1.7.5 version to 1.8.2
(typo in prev post), erased the trends records AND the trend itself so my
trends table is empty now. I had jus one trend configured - Memory-free-MB.

BR,
Eric

W dniu środa, 8 lipca 2015 14:08:45 UTC+2 użytkownik ssh...@redhat.com
napisał:

··· > > Great! I got the file. > > If it's not much trouble, can you export the trends table too? It will > save me a lot of work... > > Thanks, > Shimon. > > > On Wednesday, July 8, 2015 at 10:54:35 AM UTC+3, Ziwi wrote: >> >> Hi Simon, >> >> I've attached the recent 500k records as you requested. >> >> FYI, my delete query erased 50M records, leaving 45M, but the problem >> with high load still exists. I'll try to delete the rest of records, then >> turn off trends just to be sure this is the cause. >> >> Best regards, >> Eric >> >> W dniu wtorek, 7 lipca 2015 16:08:03 UTC+2 użytkownik ssh...@redhat.com >> napisał: >>> >>> Thanks Eric, it will be fantastic! >>> >>> On Tuesday, July 7, 2015 at 4:58:45 PM UTC+3, Ziwi wrote: >>>> >>>> Simon, >>>> >>>> Unfortunately I've already issued DELETE query for rows before 1st >>>> June. Foreman is working in production since March, so I will see how many >>>> rows will remain and I if it will be a suitable amount I'll upload them for >>>> you. >>>> >>>> Best regards, >>>> Eric >>>> >>>> W dniu wtorek, 7 lipca 2015 15:39:37 UTC+2 użytkownik ssh...@redhat.com >>>> napisał: >>>>> >>>>> Hi Eric, >>>>> >>>>> Can you upload somewhere public the most recent 500K records? >>>>> >>>>> It would help me a lot with testing a real world data to see how much >>>>> do I improve the situation. >>>>> >>>>> Thanks, >>>>> Shimon. >>>>> >>>>> >>>>> >>>>> On Tuesday, July 7, 2015 at 4:23:59 PM UTC+3, Ziwi wrote: >>>>>> >>>>>> Hello Shimon, >>>>>> >>>>>> 1. I am using Foreman 1.8.5, updated two days ago from 1.8.2 >>>>>> 2. The row count is 95087572, so massive >>>>>> >>>>>> I'm glad that you will be optimizing the trends in 1.9, however I'd >>>>>> rather not update my Foreman instance to the beta version since it's a >>>>>> production environment. So that leaves me with clearing old trends from db, >>>>>> just like you said. >>>>>> >>>>>> Thanks for your answers guys, I'll try this and see if the high load >>>>>> problem will appear again. >>>>>> >>>>>> Best regards, >>>>>> Erick >>>>>> >>>>>> W dniu wtorek, 7 lipca 2015 14:45:19 UTC+2 użytkownik >>>>>> ssh...@redhat.com napisał: >>>>>>> >>>>>>> Hi Erick, >>>>>>> >>>>>>> 1. What version of Foreman are you running? >>>>>>> 2. Can you please check the row count for trend_counters table >>>>>>> ('SELECT count(*) FROM trend_counters")? >>>>>>> >>>>>>> If my hunch is right, you will see very large row count there. If >>>>>>> this is the case I have two things for you. >>>>>>> First: I did some changes to the way the data is stored in this >>>>>>> table, so in 1.9 beta it should be more compact. >>>>>>> Second, The table stores data points from the beginning of the time, >>>>>>> so you can delete old records. ("DELETE FROM trend_counters WHERE >>>>>>> created_at < ...") >>>>>>> >>>>>>> Don't forget to back up your data before those database >>>>>>> manipulations!!! >>>>>>> >>>>>>> Shimon >>>>>>> >>>>>>> On Tuesday, July 7, 2015 at 1:24:44 PM UTC+3, Ziwi wrote: >>>>>>>> >>>>>>>> Hello Foreman users, >>>>>>>> >>>>>>>> I have a problem with a very high load on Foreman server (around >>>>>>>> 30.0). There are numerous processess with trends:counter during this high >>>>>>>> load. I've tried to stop Foreman, Foreman-proxy or even Puppet, kill all >>>>>>>> the processess with trends:counter, but they keep on popping up. >>>>>>>> >>>>>>>> I've got one trend: Memory-free-MB. The GUI for the trends works >>>>>>>> awfully slow, and when I click the delete option for this trend I get >>>>>>>> timeouted after ~20 minutes. I've also tried the command-line (foreman-rake >>>>>>>> trends:clean) but that does not help. The problem can be related to >>>>>>>> postgresql server, because the trends:counter process keeps querying DB. >>>>>>>> I've tried to vacuum PSQL but it didn't improve anything at all. >>>>>>>> >>>>>>>> Is there any way I can remove the trends manually (either from DB, >>>>>>>> or by URL, or command-line)? Or maybe there is a way to optimize postgresql >>>>>>>> database so the trends would work properly? Or my high load is related to >>>>>>>> some other thing (so how to check it). I would appreciate any help, and if >>>>>>>> you need more information about my environment please post. >>>>>>>> >>>>>>>> Thanks in advance, >>>>>>>> Erick >>>>>>>> >>>>>>>