We're glad to be on a path of getting the Foreman tasks into the foreman core
in [1] for Foreman 1.7, and believe many areas will benefit from this move,
starting with bulk actions, multi-host orchestrations (started in staypuft),
asynchronous imports, to even host provisioning (although this is not the 1.7
material probably: one step at a time).
One of the limitations foreman-tasks has at this moment it not supporting multi-host
setups. We would like to get rid of this limitation and I would appreciate your feedback
on this.
The problem and possible solutions are described here [2]. The main question is
what "medium" should be used for exchanging events between web servers and executors,
with a database on one end and messaging on the other.
I'm personally quite reluctant in introducing additional dependencies without
measurable benefit, therefore I would personally go for the database-backed solution
for start, and introducing the messaging once there are cases that
justify it. But maybe there are already plans for introducing some kind of messaging,
in which case we wound not have to concern about foreman tasks introducing the complexity
> Hey,
>
> We're glad to be on a path of getting the Foreman tasks into the foreman
> core in [1] for Foreman 1.7, and believe many areas will benefit from this
> move, starting with bulk actions, multi-host orchestrations (started in
> staypuft), asynchronous imports, to even host provisioning (although this
> is not the 1.7 material probably: one step at a time).
Good start.
> One of the limitations foreman-tasks has at this moment it not supporting
> multi-host setups. We would like to get rid of this limitation and I would
> appreciate your feedback on this.
>
> The problem and possible solutions are described here [2]. The main question
> is what "medium" should be used for exchanging events between web servers
> and executors, with a database on one end and messaging on the other.
>
> I'm personally quite reluctant in introducing additional dependencies
> without measurable benefit, therefore I would personally go for the
> database-backed solution for start, and introducing the messaging once
> there are cases that justify it. But maybe there are already plans for
> introducing some kind of messaging, in which case we wound not have to
> concern about foreman tasks introducing the complexity
I totally agree with simple way - using DB for start. We can always add
messaging later if we realize DB to be a bottleneck. While I see benefits of
messaging (delegation of delivery assurance, performance) I don't see a major
issue with shared DB which seems much easier to me.
It would be interesting to hear from others that have experience with large
setups whether and how they load-balance DB.
···
On Thursday 25 of September 2014 11:38:45 Ivan Necas wrote:
Option b (shared db for external events + local sockets) sounds like the
best choice here. We already deploy a db so that makes things a tad
easier…
How about using websockets as the only medium? I don't know how much
reliability and consistency we can get with that, hope someone more
knowledgeable on this can chime in.
···
On Fri, Sep 26, 2014 at 8:34 AM, Marek Hulan wrote:
Hi,
my 2c in text below
On Thursday 25 of September 2014 11:38:45 Ivan Necas wrote:
Hey,
We’re glad to be on a path of getting the Foreman tasks into the foreman
core in [1] for Foreman 1.7, and believe many areas will benefit from
this
move, starting with bulk actions, multi-host orchestrations (started in
staypuft), asynchronous imports, to even host provisioning (although this
is not the 1.7 material probably: one step at a time).
Good start.
One of the limitations foreman-tasks has at this moment it not supporting
multi-host setups. We would like to get rid of this limitation and I
would
appreciate your feedback on this.
The problem and possible solutions are described here [2]. The main
question
is what “medium” should be used for exchanging events between web servers
and executors, with a database on one end and messaging on the other.
I’m personally quite reluctant in introducing additional dependencies
without measurable benefit, therefore I would personally go for the
database-backed solution for start, and introducing the messaging once
there are cases that justify it. But maybe there are already plans for
introducing some kind of messaging, in which case we wound not have to
concern about foreman tasks introducing the complexity
I totally agree with simple way - using DB for start. We can always add
messaging later if we realize DB to be a bottleneck. While I see benefits
of
messaging (delegation of delivery assurance, performance) I don’t see a
major
issue with shared DB which seems much easier to me.
It would be interesting to hear from others that have experience with large
setups whether and how they load-balance DB.
I agree with all of you - not introducing another stack is better.
Evolution instead of revolution.
> How about using websockets as the only medium? I don't know how much
> reliability and consistency we can get with that, hope someone more
> knowledgeable on this can chime in.
WebSockets indeed are messaging platform, but the big difference is this
is server-client architecture. We'd need to introduce some kind of
broker anyway or create some peer-to-peer topology which I am not sure
if it what we want to build.
I lean towards RDBM with adapter design pattern. We can change our minds
later.
>
> I agree with all of you - not introducing another stack is better.
> Evolution instead of revolution.
>
> > How about using websockets as the only medium? I don't know how much
> > reliability and consistency we can get with that, hope someone more
> > knowledgeable on this can chime in.
>
> WebSockets indeed are messaging platform, but the big difference is this
> is server-client architecture. We'd need to introduce some kind of
> broker anyway or create some peer-to-peer topology which I am not sure
> if it what we want to build.
In fact, that wouldn't have to be web sockets, just plain old tcp stockets,
with a tls, which is not that hard to do with plain ruby or better event-machine.
That would however still lack the reliability, although the dynflow recovery
mechanisms don't make it that serious issue, since there is pretty much
everything else serialized, so one could still recover quite easily even when
some message is lost.
And you're right: we would end up with some kind of peer-to-peer topology,
and I believe we have time for getting to this later, or we would get to
something like AMQP at that point.
>
> I lean towards RDBM with adapter design pattern. We can change our minds
> later.
I agree: for now it seems we will start with RDBM based events sharing behind
abstraction and add other methods as optimization.
I think you two are talking about different aspects. I think websockets
is client <=> server and tcp (+tls) is server <=> server.
In the former case that sounds good. How will you handle reconnects? In
case of a load balancer you could end up with a different webserver.
Will that be able to fully retrieve the state?
In the latter case I'd try not to reinvent the wheel and go to AMQP
sooner rather than later. In the short term using the database might
suffice, but message queues were invented for this.
···
On Fri, Sep 26, 2014 at 11:35:26AM -0400, Ivan Necas wrote:
>
>
> ----- Original Message -----
> >
> > I agree with all of you - not introducing another stack is better.
> > Evolution instead of revolution.
> >
> > > How about using websockets as the only medium? I don't know how much
> > > reliability and consistency we can get with that, hope someone more
> > > knowledgeable on this can chime in.
> >
> > WebSockets indeed are messaging platform, but the big difference is this
> > is server-client architecture. We'd need to introduce some kind of
> > broker anyway or create some peer-to-peer topology which I am not sure
> > if it what we want to build.
>
> In fact, that wouldn't have to be web sockets, just plain old tcp stockets,
> with a tls, which is not that hard to do with plain ruby or better event-machine.
> That would however still lack the reliability, although the dynflow recovery
> mechanisms don't make it that serious issue, since there is pretty much
> everything else serialized, so one could still recover quite easily even when
> some message is lost.
>
> And you're right: we would end up with some kind of peer-to-peer topology,
> and I believe we have time for getting to this later, or we would get to
> something like AMQP at that point.
> In the former case that sounds good. How will you handle reconnects? In
> case of a load balancer you could end up with a different webserver.
> Will that be able to fully retrieve the state?
Websockets and TCP sockets are indeed out of the table. The discussion
is RDBM vs AMQP. And I say, as a big messaging advocate, to stick with
RDBM for the first phase.
I am pretty sure RDBM will likely fulfill all the requirements we will
eventually have. AMQP wins in the distributed environments, but Foreman
app (the web ui) is centralized solution (one main DB).
> > In the former case that sounds good. How will you handle reconnects? In
> > case of a load balancer you could end up with a different webserver.
> > Will that be able to fully retrieve the state?
>
> Websockets and TCP sockets are indeed out of the table. The discussion
> is RDBM vs AMQP. And I say, as a big messaging advocate, to stick with
> RDBM for the first phase.
>
> I am pretty sure RDBM will likely fulfill all the requirements we will
> eventually have. AMQP wins in the distributed environments, but Foreman
> app (the web ui) is centralized solution (one main DB).
+1 Dynflow can be pretty useful for orchstrating AMQP events, but it itself
is not the best fit for running on it. Btw. I'm looking forward to try
this for getting rid of some frequent polling on db table http://www.postgresql.org/docs/8.4/static/sql-notify.html (of course we will
keep support for other dbs, but that doesn't mean we can't use the additional
features when available)
