Is anyone running Foreman with Galera? I've had my lab environment (1.13.3) running on a 3 node MariaDB 10.1 Galera cluster for the past few weeks and haven't noticed any issues thus far. I haven't seen anything in the way of "official/listed" support, and reading into Galera, there "should" be no issues other than if Foreman didn't support innodb, or used tables without keys. I've also googled a bit and found a few threads that seemed to indicate "it should work" and "let us know if it doesn't" but they are older posts at this point.
That begs my question, is anyone using it? Long-term? Any issues, gotchas? Most importantly, is there any reason I shouldn't use it in my production environment? Our foreman environment currently has ~8600 servers reporting in which translates to quite a few DB calls during normal operation. We've been using mysql's built in replication until now - but are looking to upgrade to using Galera if possible for better seamless ha/failover between our 3 datacenters.
Finally, if it's a bad idea, I'm definitely open to suggestions/advice on an HA/load balanced database solution that might be better suited for Foreman?
Thanks for any and all feedback, advice, and suggestions
~Jason Lang
The information contained in this message may be privileged, confidential and protected from disclosure. If the reader of this message is not the intended recipient, or an employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this communication in error, please notify your representative immediately and delete this message from your computer. Thank you.
I would like to know as well as I'm still running PgSQL locally and want to
move that to one of my MySQL clusters running Percona.
As far as I know it should not be any problem, have you tested it already ?
Thanks,
Matt
···
Op zondag 2 april 2017 15:40:30 UTC+2 schreef Lang, Jason:
>
> Hey All,
>
>
>
> Is anyone running Foreman with Galera? I’ve had my lab environment
> (1.13.3) running on a 3 node MariaDB 10.1 Galera cluster for the past few
> weeks and haven’t noticed any issues thus far. I haven’t seen anything in
> the way of “official/listed” support, and reading into Galera, there
> “should” be no issues other than if Foreman didn’t support innodb, or used
> tables without keys. I’ve also googled a bit and found a few threads that
> seemed to indicate “it should work” and “let us know if it doesn’t” but
> they are older posts at this point.
>
>
>
> That begs my question, is anyone using it? Long-term? Any issues, gotchas?
> Most importantly, is there any reason I shouldn’t use it in my production
> environment? Our foreman environment currently has ~8600 servers reporting
> in which translates to quite a few DB calls during normal operation. We’ve
> been using mysql’s built in replication until now – but are looking to
> upgrade to using Galera if possible for better seamless ha/failover between
> our 3 datacenters.
>
>
>
> Finally, if it’s a bad idea, I’m definitely open to suggestions/advice on
> an HA/load balanced database solution that might be better suited for
> Foreman?
>
>
>
> Thanks for any and all feedback, advice, and suggestions
>
>
>
> ~Jason Lang
>
> The information contained in this message may be privileged, confidential
> and protected from disclosure. If the reader of this message is not the
> intended recipient, or an employee or agent responsible for delivering this
> message to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> your representative immediately and delete this message from your computer.
> Thank you.
>
We evaluate the option but later we figure it out then as we needed
Katello we still needed pg.
We never tested Foreman with Galera on the field.
···
On Sunday, April 2, 2017 at 3:40:30 PM UTC+2, Lang, Jason wrote:
>
> Hey All,
>
>
>
> Is anyone running Foreman with Galera? I’ve had my lab environment
> (1.13.3) running on a 3 node MariaDB 10.1 Galera cluster for the past few
> weeks and haven’t noticed any issues thus far. I haven’t seen anything in
> the way of “official/listed” support, and reading into Galera, there
> “should” be no issues other than if Foreman didn’t support innodb, or used
> tables without keys. I’ve also googled a bit and found a few threads that
> seemed to indicate “it should work” and “let us know if it doesn’t” but
> they are older posts at this point.
>
>
>
> That begs my question, is anyone using it? Long-term? Any issues, gotchas?
> Most importantly, is there any reason I shouldn’t use it in my production
> environment? Our foreman environment currently has ~8600 servers reporting
> in which translates to quite a few DB calls during normal operation. We’ve
> been using mysql’s built in replication until now – but are looking to
> upgrade to using Galera if possible for better seamless ha/failover between
> our 3 datacenters.
>
>
>
> Finally, if it’s a bad idea, I’m definitely open to suggestions/advice on
> an HA/load balanced database solution that might be better suited for
> Foreman?
>
>
>
> Thanks for any and all feedback, advice, and suggestions
>
>
>
> ~Jason Lang
>
> The information contained in this message may be privileged, confidential
> and protected from disclosure. If the reader of this message is not the
> intended recipient, or an employee or agent responsible for delivering this
> message to the intended recipient, you are hereby notified that any
> dissemination, distribution or copying of this communication is strictly
> prohibited. If you have received this communication in error, please notify
> your representative immediately and delete this message from your computer.
> Thank you.
>
It's not a bad idea, but there's a lot that comes to mind you will want to
be careful with. I have ran non-foreman apps on Galera for years with great
success and quite serious scale (the biggest is currently running 1.4M QPS
across a 6 node cluster). A random list:
Make sure you have sufficient performance capacity across the cluster.
When a node can't keep up with the cluster it goes into a state called flow
control, which essentially stops all writes cluster wide until the slow
node has caught up. I don't need to explain you will want to tune and make
sure flow control never gets triggered.
the real power of Galera comes when you can loadbalance reads across the
cluster, but this means the app needs to support intelligent multiple
connection strings. It's a long story, but I'll try to be short:
** In Galera you can in theory send any connection to any node and it will
have a consistent state (semi-sync replication). In reality, a node can be
behind a few contracts and this can lead to reads happening directly after
a write not finding what they just wrote. The other problem is hotspots. If
many places in the application write to the same database page but those
connections go to different nodes in the cluster you can get (essentially)
deadlock errors.
** to counter the above symptoms, most custom applications use something
like a haProxy load balancer in tcp mode that has two frontends: 1 for
writes and reads that need to happen directly after writes, the second for
all other selects. However, I'm not aware of the foreman supporting such a
thing at present. That means you have two options depending on how much you
like to live on the edge: 1) you send all foreman traffic straight up to a
single node in the cluster, thereby losing scalability for the foreman app.
you send all connections load balanced to all galera nodes (using
haproxy if you want) and you cross your fingers that it will all work out.
So, all of the above kept in mind, I'd totally give it a go (Galera beats
Async replication on many fronts). What would I do to be on the safe side?
Set up a galera cluster and make the whole cluster a slave of your
existing mysql master. Let that run for a few weeks and monitor closely so
you know it can run at least your write load properly
Presuming you have a test environment set up another cluster there and
take some access logs from production and replay them with a load testing
tool to see how that holds up. Once happy do the same on your production
galera cluster.
Set up haproxy as mentioned above with two frontends, but send all
traffic to the frontend that goes to a single node in the cluster
Start monitoring where your heaviest/most frequent selects are coming
from and hack foreman to send those to your secondary haproxy frontend
which loadbalances round robin.
···
On Sunday, April 2, 2017 at 11:54:45 PM UTC+3, Matt wrote:
>
> I would like to know as well as I'm still running PgSQL locally and want
> to move that to one of my MySQL clusters running Percona.
>
> As far as I know it should not be any problem, have you tested it already ?
>
> Thanks,
>
> Matt
>
> Op zondag 2 april 2017 15:40:30 UTC+2 schreef Lang, Jason:
>>
>> Hey All,
>>
>>
>>
>> Is anyone running Foreman with Galera? I’ve had my lab environment
>> (1.13.3) running on a 3 node MariaDB 10.1 Galera cluster for the past few
>> weeks and haven’t noticed any issues thus far. I haven’t seen anything in
>> the way of “official/listed” support, and reading into Galera, there
>> “should” be no issues other than if Foreman didn’t support innodb, or used
>> tables without keys. I’ve also googled a bit and found a few threads that
>> seemed to indicate “it should work” and “let us know if it doesn’t” but
>> they are older posts at this point.
>>
>>
>>
>> That begs my question, is anyone using it? Long-term? Any issues,
>> gotchas? Most importantly, is there any reason I shouldn’t use it in my
>> production environment? Our foreman environment currently has ~8600 servers
>> reporting in which translates to quite a few DB calls during normal
>> operation. We’ve been using mysql’s built in replication until now – but
>> are looking to upgrade to using Galera if possible for better seamless
>> ha/failover between our 3 datacenters.
>>
>>
>>
>> Finally, if it’s a bad idea, I’m definitely open to suggestions/advice on
>> an HA/load balanced database solution that might be better suited for
>> Foreman?
>>
>>
>>
>> Thanks for any and all feedback, advice, and suggestions
>>
>>
>>
>> ~Jason Lang
>>
>> The information contained in this message may be privileged, confidential
>> and protected from disclosure. If the reader of this message is not the
>> intended recipient, or an employee or agent responsible for delivering this
>> message to the intended recipient, you are hereby notified that any
>> dissemination, distribution or copying of this communication is strictly
>> prohibited. If you have received this communication in error, please notify
>> your representative immediately and delete this message from your computer.
>> Thank you.
>>
>
