>
> > I'm finishing adding bulk actions to Foreman Tasks these days (there are
> > couple of bulk actions in Katello that need exactly the same treatment).
> > I can do a deep-dive on this later on.
> >
> > I'm not a big fan of having two ways to deal with the same scenario
> > and deciding which on to use based on the load. There are other benefits
> > for using the Foreman Tasks for deleting just 2 hosts (and even 1,
> > especially
> > when the orchestration process itself is managed by Dynflow, said that,
> > it's
> > possible to just call system.destroy! in the plan phase and let the current
> > orchestration to handle the stuff, so that we could start using Foreman
> > Tasks
> > even with the old orchestration in place): tracking the status of the bulk
> > action
> > and having a record about this in the tasks for auditing.
> >
> > I've started seeing Foreman Tasks/Dynflow being referenced in more
> > discussions
> > lately: remote execution, multi-host deployments used in Staypuft, better
> > error-handling in host orchestration, bulk actions. It maybe makes sense to
> > start thinking about adopting foreman-tasks as a core foreman feature and
> > being
> > able to rely on it's presence.
>
>
> > I can think about four things to be added to Dynflow or Foreman Tasks
> > to support the host creation orchestration:
> >
> > 1. support for multi-host deployments: right now, we rely on the Dynflow
> > executre
> > running on the same host as the foreman server. This is a known
> > limitation
> > we have to solve sooner or later. We have ideas on how to solve that,
> > it's
> > more about prioritization.
> >
> What is the actual requirement to run it on the same host? I assume that if
> its another server with the code / access to db it should just work?
The thing is the Dynflow supports suspending actions and waking them up
on events: for example when doing polling, we suspend the action and plan
to send an event to wake it up and perform another polling in some interval.
Between the two events, the action is not consuming thread, letting other work
to be done.
The events are not limited just for the polling, in staypuft, there are events
triggered after provisioning is finished, so the event is produced thought the
Foreman API (/unattended/built action), and we need to propagate the event to
the executor. The same applies to canceling of the tasks.
The communication goes also the other direction: you can trigger the task on web server
and wait for it to finish in the very same request: the executor after the task finishes
sends a message back to the web server to continue working (we don't do polling)
In one-host setup, we use a unix socket for the web-server -> executor communication,
since it requires minimal setup. For multi-host setup, it seems we would need
other form of communication (probably messaging) for the executors and web servers
to talk to each other.
>
> >
> > 2. rollbacks support: in the Katello's usage, using the task resume
> > is the one to go for majority of the cases. It however makes a lot of
> > sense
> > in host provisioning. Relying on Dynflow in the orchestration would
> > allow
> > us to decide if for particular error, it's better to rollback or
> > or let the admin to fix the external issue and continue with the process
> >
> true, atm, most (all?) of the host orchestration are serial tasks, meaning
> you need the output of one to move on to the next step, leaving things like
> a blank vm entry is not really useful.
>
> further, the real technical reason imho, is how to present failures to the
> user, and I would prefer in the first adoption phase to leave the host
> orchestration for a future consolidation attempts.
Not doing everything at once is of course better. As you said, the first candidates
would be the steps that take a while to be executed, such as ssh finish scripts.
I wonder, if the ssh-finish scripts shouldn't be considered as a form of remote execution
rather than limiting that for just provisioning.
>
> >
> > 3. backward compatibility with the old orchestration: ideally, there should
> > be a way how run the current orchestration steps as part of a Dynflow
> > tasks.
> > Although, the core orchestration would be still benefit more from
> > transferring
> > to dynflow actions, we need a strategy on how to keep third-party
> > plugins
> > (including Foreman hooks) working.
> >
> > 4. On the Foreman side, it would require to drop the Ruby 1.8.7 support.
> >
>
> > Just for the bulk actions, the (1) and (4) should be sufficient to
> > unblock us from proceeding. The (2) and (3) would be required for
> > provisioning
> > orchestration.
> >
> > I believe Foreman would benefit a lot from the adoption (I'm might be a
> > little
> > biased for obvious reasons though), at least we got really good feedback
> > so far when using it for Katello orchestration. The Dynflow itself is also
> > moving forward, with the pitr-ch improved actors implementation getting
> > merged into concurrent-ruby gem [1] (we have a basic one right now
> > directly in Dynflow).
> > We will switch to the concurrent-ruby soon, so that we can share the
> > concurrent stuff
> > with others: it's getting really good feedback so far.
> >
> > So the questions I have are:
> >
> > 1. is the Foreman team willing at some point to merge
> > foreman-tasks into the core (or make it a hard dependency). What are
> > the other things
> > that need to be done before being able to do so?
> >
> I would assume mostly installer, upgrade, docs etc ? potentially a review
> of foreman-tasks code itself?
Sounds good to me.
>
>
> > 2. if (1) holds, what version it would be for (it would need dropping the
> > ruby 1.8.7
> > for that version)?
> >
> IMHO its time, can we target it for 1.7 release?
That would be fantastic.
>
>
> > 3. would it be possible to use Dynflow for the host orchestration? I'm
> > sure it would be
> > possible from the Dynflow point of view (we have a lot of experience
> > with doing so
> > in the Katello codebase), but I'm aware that the orchestration has a
> > long
> > history and people might be defensive on rewriting that.
> >
> I have no objections, but I see smaller value in that vs adding async
> support for long running tasks (e.g. puppet class imports, report
> processing, bulk actions etc).
>
> we could consider extracting really long tasks (such as SSH into an
> instance).
+1
···
----- Original Message -----
> On Tue, Aug 12, 2014 at 1:07 PM, Ivan Necas wrote:
The answers for this questions would help us better prioritize the things
to keep
things going forward.
[1] - https://github.com/ruby-concurrency/concurrent-ruby
Cheers,
thanks Ivan!
Ohad
Ivan
----- Original Message -----
I was wondering, how useful would it be to take advantage of
Foreman-Tasks
and use it to support many or heavy-to-process requests?
I’m thinking the user could set a threshold per Foreman host depending on
their processing/memory and we make requests be handled by Foreman-tasks
once requests go over that.
For instance, deleting 100 hosts is a task bound to timeout or fail
because
of all the associated data (reports/logs, etc)
Your thoughts?
–
Daniel Lobato
@elobatoss
blog.daniellobato.me
<
https://mailtrack.io/trace/link/657886dff180af4370c51f673bcde0ece76cba93>
daniellobato.me
<
https://mailtrack.io/trace/link/1b6e7a141b5dd01c9f62582aea5ae796d4d88d57>
GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.