I'm working on an HA architecture for Foreman and have been experimenting
w/ pointing multiple Foreman instances at the same MySQL database. I
expected to find issues early but despite my (limited) testing everything
seems to be working fine. Assuming I'm not using Puppet and other services
like DHCP and TFTP are hosted elsewhere, could someone explain why this
should not work? I'm sure there are quite a few reasons.
-Chris
It's meant to work like this, just a scalable web app backing onto a
database.
There are a couple of things to be wary of though. One is the Rails
cache, usually stored on disk in tmp/ - it's used for various things in
Foreman, but inconsistencies could cause weird problems. The
foreman_memcache plugin is the usual solution, and changes the cache to
store data in memcached, so all servers share one cache.
The other is the encryption key stored in
/etc/foreman/encryption_key.rb, which needs to be the same on all
servers (used for compute resources and LDAP passwords).
We don't have any documentation on this kind of setup though, so if you
do get the time, I'd love to see this knowledge added to the manual.
–
Dominic Cleal
Red Hat Engineering
I run foreman 1.8.2 on two nodes right now as I have about 50 power users &
1600 nodes and wanted something robust. I did the changes that Dominic
spoke of, installed the memcache plugin and use that and also setup the
/etc/foreman/encryption_key.rb to be the same. Our instances run behind
citrix netscaler and i tell the netscaler to not round robin load balance
but distribute load and preserve session to hosts.
The biggest Issue i had was trying to have foreman work out of the box with
installers and puppet private certs while allowing external users to have a
valid SSL cert to access foreman so they wouldn't get browser warnings (or
in safari's case, completely blocked)… it was a little weird getting this
to work on netscaler…
Another thing I had issues with was the websockets cert keeps changing in
audit log because two nodes are trying to update that with their respective
certs… there was one other thing i ran into with proxies but i don't
remember how/what i fixed… people were complaining that one node or the
other couldn't view the status of the proxies (would show its down) and
would randomly get warnings that one of the app tiers didn't work… i'm
thinking hard about this and it may have been i had to re-generate the SSL
certs to have an alt name across the board for all nodes for the foreman
url we use to get in… (which was part of the quirks of getting it behind a
netscaler with a real public trusted cert and not break the internal puppet
foreman cert stuff)
-byron
> > I'm working on an HA architecture for Foreman and have been
> > experimenting w/ pointing multiple Foreman instances at the same MySQL
> > database. I expected to find issues early but despite my (limited)
> > testing everything seems to be working fine. Assuming I'm not using
> > Puppet and other services like DHCP and TFTP are hosted elsewhere, could
> > someone explain why this should not work? I'm sure there are quite a few
> > reasons.
>
> It's meant to work like this, just a scalable web app backing onto a
> database.
>
> There are a couple of things to be wary of though. One is the Rails
> cache, usually stored on disk in tmp/ - it's used for various things in
> Foreman, but inconsistencies could cause weird problems. The
> foreman_memcache plugin is the usual solution, and changes the cache to
> store data in memcached, so all servers share one cache.
>
> The other is the encryption key stored in
> /etc/foreman/encryption_key.rb, which needs to be the same on all
> servers (used for compute resources and LDAP passwords).
>
another thing I'll add, even if I assume you don't use is foreman-tasks,
I'm not sure how this works in such setup maybe Ivan can expand on this?
>
> We don't have any documentation on this kind of setup though, so if you
> do get the time, I'd love to see this knowledge added to the manual.
>
> +100
Ohad
–
Dominic Cleal
Red Hat Engineering–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
> I run foreman 1.8.2 on two nodes right now as I have about 50 power users &
> 1600 nodes and wanted something robust. I did the changes that Dominic
> spoke of, installed the memcache plugin and use that and also setup the
> /etc/foreman/encryption_key.rb to be the same. Our instances run behind
> citrix netscaler and i tell the netscaler to not round robin load balance
> but distribute load and preserve session to hosts.
>
> The biggest Issue i had was trying to have foreman work out of the box with
> installers and puppet private certs while allowing external users to have a
> valid SSL cert to access foreman so they wouldn't get browser warnings (or
> in safari's case, completely blocked)… it was a little weird getting this
> to work on netscaler…
There is a setting in the installer named --puppet-dns-alt-names where
you can set a shared alt name (like foreman.example.com). This can also
help with the changing websockets certs. I've never used it in the
context of the installer and I recall that you had to manually sign the
certificate on the CA, but it's been a while since I used it altogether.
> Another thing I had issues with was the websockets cert keeps changing in
> audit log because two nodes are trying to update that with their respective
> certs… there was one other thing i ran into with proxies but i don't
> remember how/what i fixed… people were complaining that one node or the
> other couldn't view the status of the proxies (would show its down) and
> would randomly get warnings that one of the app tiers didn't work… i'm
> thinking hard about this and it may have been i had to re-generate the SSL
> certs to have an alt name across the board for all nodes for the foreman
> url we use to get in… (which was part of the quirks of getting it behind a
> netscaler with a real public trusted cert and not break the internal puppet
> foreman cert stuff)
Like I said, the alt name could help with the websockets cert. You can
also explicitly set the certificate in the settings to something that
you share on all machines. It's a different setting than the normal
certificates.
-byron
On Thursday, July 23, 2015 at 7:01:40 AM UTC-5, Chris wrote:
I’m working on an HA architecture for Foreman and have been experimenting
w/ pointing multiple Foreman instances at the same MySQL database. I
expected to find issues early but despite my (limited) testing everything
seems to be working fine. Assuming I’m not using Puppet and other services
like DHCP and TFTP are hosted elsewhere, could someone explain why this
should not work? I’m sure there are quite a few reasons.-Chris
–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
>
> > > I'm working on an HA architecture for Foreman and have been
> > > experimenting w/ pointing multiple Foreman instances at the same MySQL
> > > database. I expected to find issues early but despite my (limited)
> > > testing everything seems to be working fine. Assuming I'm not using
> > > Puppet and other services like DHCP and TFTP are hosted elsewhere, could
> > > someone explain why this should not work? I'm sure there are quite a few
> > > reasons.
> >
> > It's meant to work like this, just a scalable web app backing onto a
> > database.
> >
> > There are a couple of things to be wary of though. One is the Rails
> > cache, usually stored on disk in tmp/ - it's used for various things in
> > Foreman, but inconsistencies could cause weird problems. The
> > foreman_memcache plugin is the usual solution, and changes the cache to
> > store data in memcached, so all servers share one cache.
> >
> > The other is the encryption key stored in
> > /etc/foreman/encryption_key.rb, which needs to be the same on all
> > servers (used for compute resources and LDAP passwords).
> >
>
> another thing I'll add, even if I assume you don't use is foreman-tasks,
> I'm not sure how this works in such setup maybe Ivan can expand on this?
So in case you use foreman-tasks (using plugin that leverages that,
such as Katello, foreman-chef, foreman-salt, foreman-docker) it's good to know,
that before foreman-tasks-0.7.x, this setup was not supported. Dynflow 0.8.x (introduced
with foreman-tasks-0.7.x) added active-active support, so
the multi-hosts setup should just start working: no additional setup needed: every host can run
the foreman-tasks service and the work will be spread across the workers.
– Ivan
We don’t have any documentation on this kind of setup though, so if you
do get the time, I’d love to see this knowledge added to the manual.+100
Ohad
–
Dominic Cleal
Red Hat Engineering–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
Yeah, I used dns_alt_names to have "foreman.myhost.local" work.
And to answer the other question that was asked, I just install the cron
jobs on a single node for the time being.
I would like to see better installer support for an "inside" cert and
"external" cert… i see lots of people shooting themselves in the foot
trying to use a public SSL on web front end and breaking all the backend
puppet stuff.
> Yeah, I used dns_alt_names to have "foreman.myhost.local" work.
Using .local should be discouraged since it's reserved in RFC 6762 for
Multicast DNS. The DNS guys I talk to generally suggest using a real
domain. If security is a factor and it should be hidden, just ensure
your nameservers never return it to the outside world. (I know many
companies have policies, but it never hurts to state the best
practices.)
> And to answer the other question that was asked, I just install the cron
> jobs on a single node for the time being.
It's not pretty, but it works.
> I would like to see better installer support for an "inside" cert and
> "external" cert… i see lots of people shooting themselves in the foot
> trying to use a public SSL on web front end and breaking all the backend
> puppet stuff.
That's good feedback and I agree many users would like this. Right now I
don't know if it's supported and if so, how to configure it.
On Friday, July 24, 2015 at 2:57:12 AM UTC-5, Ewoud Kohl van Wijngaarden > wrote:
There is a setting in the installer named --puppet-dns-alt-names where
you can set a shared alt name (like foreman.example.com). This can also
help with the changing websockets certs. I’ve never used it in the
context of the installer and I recall that you had to manually sign the
certificate on the CA, but it’s been a while since I used it altogether.
We've had our internal TLD for 10+ years and we're working with the company
that now owns it…
Well aware of what we have to work through…
https://www.icann.org/news/announcement-2-2014-08-01-en
btw, the example i gave isn't exactly what we use… hehe, i'm not
intentionally leaking dns for the world to hit 