Purging facts?

I've got 200+ servers and rapidly accumulating facts. Is there a
straightforward way to purge them the way we do with reports?

(foreman 0.4.2, puppet server 2.7.11, linux OEL 6.2, mysql 14.14…)

thanks Betsy

> I've got 200+ servers and rapidly accumulating facts. Is there a
> straightforward way to purge them the way we do with reports?
>
> (foreman 0.4.2, puppet server 2.7.11, linux OEL 6.2, mysql 14.14…)
>
> thanks Betsy

You don't really want to purge facts, as they don't take a lot of
space, and are very useful in searches. Can you explain what the issue
is?

··· On Tue, Jun 5, 2012 at 1:16 PM, Betsy Schwartz wrote:


You received this message because you are subscribed to the Google Groups “Foreman users” group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en.

> You don't really want to purge facts, as they don't take a lot of
> space, and are very useful in searches. Can you explain what the issue

Because the Facts page is horribly slow. I have 200+ servers and
200,000+ facts.
(foreman 0.4.2, puppet server 2.7.11, linux OEL 6.2, mysql
14.14…vmware with an EMC backend)

It takes about half a second to open any other screen in Foreman. When
I go to the Facts screen it takes almost thirty seconds.

Obviously I need to tune mysql also :slight_smile: but I don't really need the
facts to pile up forever when we're only using the current ones.

thanks Betsy
(msql tuning hints also welcome! )

PS is there reporting that I've missed on the old facts? Can I do
something interesting with them?

> > You don't really want to purge facts, as they don't take a lot of
> > space, and are very useful in searches. Can you explain what the issue
>
> Because the Facts page is horribly slow. I have 200+ servers and
> 200,000+ facts.
> (foreman 0.4.2, puppet server 2.7.11, linux OEL 6.2, mysql
> 14.14…vmware with an EMC backend)
>
> It takes about half a second to open any other screen in Foreman. When
> I go to the Facts screen it takes almost thirty seconds.
>

thats interesting, as I don't see that here (even with a much larger
dataset).

can you provide some logs?

200,000 facts for 200 hosts sounds like way too many, and facter/foreman
should not have as many.

Do you see duplicate facts? any chance you upload more than onces? (e.g.
use storeconfigs and cron job and the new enc to upload facts)

as far as I remember, the facts merging code (which is actually part of the
original puppet storeconfigs code), removes all facts and re-add them every
time (or at least in memory), so if you upload them at the same time, you
could get duplicates.

in anycase, facts should be expired (e.g. you should see exactly the same
facts you see in the output of facter -p).

Ohad

··· On Wed, Jun 6, 2012 at 2:23 PM, Betsy Schwartz wrote:

Obviously I need to tune mysql also :slight_smile: but I don’t really need the
facts to pile up forever when we’re only using the current ones.

thanks Betsy
(msql tuning hints also welcome! )


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to
foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/foreman-users?hl=en.

Betsy, I need to look into this. Thanks, Brian

··· On Wed, Jun 6, 2012 at 7:24 AM, Betsy Schwartz wrote: > PS is there reporting that I've missed on the old facts? Can I do > something interesting with them? > > -- > You received this message because you are subscribed to the Google Groups "Foreman users" group. > To post to this group, send email to foreman-users@googlegroups.com. > To unsubscribe from this group, send email to foreman-users+unsubscribe@googlegroups.com. > For more options, visit this group at http://groups.google.com/group/foreman-users?hl=en. >

OK I figured it out - I've actually got 200,000 uniq instances of a
particular legitimate fact on 200+ machines (30+ machines that have
7000+ facts each). Jumped to a conclusion that it wasn't purging.

So now I've got to tune this better. Any suggestions on tuning mysql
for this many facts? It is painfully slow but good to know that it's
not going to get any slower.

> OK I figured it out - I've actually got 200,000 uniq instances of a
> particular legitimate fact on 200+ machines (30+ machines that have
> 7000+ facts each). Jumped to a conclusion that it wasn't purging.
>
> So now I've got to tune this better. Any suggestions on tuning mysql
> for this many facts? It is painfully slow but good to know that it's
> not going to get any slower.
>

I'm still not sure why it would be that slow, as we dont pull the entire
content, the only case where it could be slow, is if you are doing any
sorting based on hostnames etc.

can you try playing with the headers (change sort) and see if that makes
any difference?
also try to view a specific system facts, e.g. /hosts/<fqdn>/facts, is that
as slow?

Ohad

··· On Wed, Jun 6, 2012 at 6:32 PM, Betsy Schwartz wrote:


You received this message because you are subscribed to the Google Groups
"Foreman users" group.
To post to this group, send email to foreman-users@googlegroups.com.
To unsubscribe from this group, send email to
foreman-users+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/foreman-users?hl=en.

I think the issue is it's slow to initially load the 'facts' page (click on
'Facts' tab … URL /fact_values).

This is the same for me as well. If I access facts from a host, takes like
2-4 seconds (via the URL you said to try). Going to /fact_values via the
'Facts' tab takes almost 40 seconds every time.

Regards,
Jake

Sorting does take as long as calling up the original facts page.
Anywhere from 30-60 seconds.
Searching for facts from one host takes 5-10 seconds.
Server load is otherwise low.

  1. which db is that
  2. can you turn on debug and provide the raw log? (should include sql and
    timing etc).

once you find the query (or queries) that take a real long time, ideally
send the output of something like with explain statement.

thanks!
Ohad

··· On Wed, Jun 6, 2012 at 7:48 PM, Betsy Schwartz wrote:

Sorting does take as long as calling up the original facts page.
Anywhere from 30-60 seconds.
Searching for facts from one host takes 5-10 seconds.
Server load is otherwise low.

db is mysql
will look into debug this weekend, thanks!

··· On Wed, Jun 6, 2012 at 3:00 PM, Ohad Levy wrote: > > > On Wed, Jun 6, 2012 at 7:48 PM, Betsy Schwartz > wrote: >> >> Sorting does take as long as calling up the original facts page. >> Anywhere from 30-60 seconds. >> Searching for facts from one host takes 5-10 seconds. >> Server load is otherwise low. >> > > 1. which db is that > 2. can you turn on debug and provide the raw log? (should include sql and > timing etc). > > once you find the query (or queries) that take a real long time, ideally > send the output of something like with explain statement. > > thanks! > Ohad > > -- > You received this message because you are subscribed to the Google Groups > "Foreman users" group. > To post to this group, send email to foreman-users@googlegroups.com. > To unsubscribe from this group, send email to > foreman-users+unsubscribe@googlegroups.com. > For more options, visit this group at > http://groups.google.com/group/foreman-users?hl=en.