Rollback mechanism in dynflow?

#5863 - dynflow rollback needed when content host creation fails in either candlepin or pulp step
http://projects.theforeman.org/issues/5863

What is the suggested mechanism for rolling back a failed dynflow? Is there a way to specify rollbacks for each action?

··· -- @thomasmckay


“The leader must aim high, see big, judge widely, thus setting himself apart form the ordinary people who debate in narrow confines.” ~ Charles De Gaulle

“Leadership is about making others better as a result of your presence and making sure that impact lasts in your absence.” ~ Harvard Business School

> #5863 - dynflow rollback needed when content host creation fails in either
> candlepin or pulp step
> Feature #5863: dynflow rollback needed when content host creation fails in either candlepin or pulp step - Katello - Foreman
>
> What is the suggested mechanism for rolling back a failed dynflow? Is there a
> way to specify rollbacks for each action?

We have the rollback feature on our TODO for dynflow (it's on the README as well
define rollback for the workflow (planned)).

However, rolling back a transaction that goes outside of your local database
is a tricky thing and it also might not be possible at all (e.g. resource deletion)
or might not be reliable enough.
Therefore, even when the definition of rollback will be possible in Dynflow (I hope
to have time allocated for doing that soon, please keep shouting that we need that:),
I would still highly recommend eliminating the known error states in the plan
phase.

Of course, there is a risk of race conditions, so still some checks in plan phase
might pass and the state occurring just in the run phase, and for this cases the
rollback is one possibility of dealing with that (together with other possibilities).

For now, we've mainly focused on the happy path with fallback to resuming/skipping steps.
More options of handling the error states should be on the plan for the next iterations.

– Ivan

··· ----- Original Message -----


@thomasmckay


“The leader must aim high, see big, judge widely, thus setting himself apart
form the ordinary people who debate in narrow confines.” ~ Charles De Gaulle

“Leadership is about making others better as a result of your presence and
making sure that impact lasts in your absence.” ~ Harvard Business School


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Rollbacks is wrong concept in Katello. One can only rollback in a
system that has (distributed) transactions. This is not the case for
Katello, therefore it is much easier and comfortable to use transaction
compensation mechanisms together with dynflow capabilities to suspend
failed process.

One can always go back and recover from error manually, for actions that
can fail more often a compensation mechanism can be coded, which is
essentially another workflow/process that put things back in order. Like
a rollback.

I think rollbacks is a complication and one can live without them
easily.

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman

> Rollbacks is wrong concept in Katello. One can only rollback in a
> system that has (distributed) transactions. This is not the case for
> Katello, therefore it is much easier and comfortable to use transaction
> compensation mechanisms together with dynflow capabilities to suspend
> failed process.
>
> One can always go back and recover from error manually, for actions that
> can fail more often a compensation mechanism can be coded, which is
> essentially another workflow/process that put things back in order. Like
> a rollback.

That's exactly the plan! Nicely said.

– Ivan

··· ----- Original Message -----

I think rollbacks is a complication and one can live without them
easily.


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

> Rollbacks is wrong concept in Katello. One can only rollback in a
> system that has (distributed) transactions. This is not the case for
> Katello, therefore it is much easier and comfortable to use transaction
> compensation mechanisms together with dynflow capabilities to suspend
> failed process.
>
> One can always go back and recover from error manually, for actions that
> can fail more often a compensation mechanism can be coded, which is
> essentially another workflow/process that put things back in order. Like
> a rollback.

Agreed. Please substitute "compensate" for "rollback." :slight_smile:

So, what I'm suggesting is a defined way in dynflow to compensate for failures. For example, several steps are run in parallel so the compensate() method would need to know which of those steps succeeded/failed and then do appropriate recovery steps.

··· ----- Original Message -----

I think rollbacks is a complication and one can live without them
easily.


Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

> So, what I'm suggesting is a defined way in dynflow to compensate for failures. For example, several steps are run in parallel so the compensate() method would need to know which of those steps succeeded/failed and then do appropriate recovery steps.

If this can be generic, that would be great. Compensation calls should
be standalone processes (or actions) themselves. But it is usually very
challenging to automate that. For the first phase, I'd recommend just to
have those processes/workflows ready to be consumed manually.

I learned one thing: compensation processes fail too :slight_smile:
It is also good to have the ability to manually modify data pipeline and
fix them manually when necessary.

Can I manually initiate a process/workflow in foreman-tasks/dynflow from
the console? This would be great tool for recovery and also for
initiating compensation processes. Should be maybe enough for the first
cut.

··· -- Later,

Lukas “lzap” Zapletal
irc: lzap #theforeman