Foreman At Large/Huge Scale?

Hey Matt,

Generally I agree that lowering my check-in to 2-3 hours would greatly alleviate issues. I’ve stubbornly “refused” because i like the general compliance of 30 minutes vs 3 hours, and my issues at this point are only annoyances. Foreman still “woks” but that is my last resort!

I haven’t seen performance issues with smartclass param(s) other than the 1.18 bug which thankfully was fixed :slight_smile:

Regarding the upgrade slowness - how many “Foreman” servers do you have? With just one, it’s less noticeable. But with many - the upgrade time is multiplied per-server, and you can’t really do many in parallel as most of the queries are blocking/locking, causing them to just queue for hours if many nodes are doing the migrate/seed tasks as part of the RPM post-install tasks…

I can also say class import is slow (15-30 minutes for us currently, and often taking several tries due to hitting the (too low) timeout on 1 environment of 30 causing the whole thing to bomb :frowning:

We reduced our audit page timeouts by clearing the audit log(s) via the audits:clean rake task - but we can now only keep like 1 month (which is still something like 250k rows in a table) - this lets the audit pages “load” but still fairly slowly…

Fact searches are super slow, but recently we have also seen the Parameter search become extremely slow. my smaller test environment can still return a detailed param search in 2-3 second(s) but my prod environment simply never finishes complex param searches anymore I’m hopeful that this “may” be an index/optimization issue specific to mysql - but not really sure at this point: Host Params API Lookups Timeout and will not return has more details/specifics on this as observed on my end.

We haven’t ever gotten the Puppet CA Proxy Page to load - we just use the puppet cert CLI command(s) when necessary - which is barely ever :slight_smile: