Moving foreman-tasks to the core: the plan

[iNecas]

Hello friends,

You've might noticed we started using foreman-tasks inside the
foreman-core since [1] was merged. Unfortunately, we hit issues in the
packaging phase and the proposed workarounds were not accepted ([2]
[3]).

Based on various discussions, as well as on the fact that
foreman-tasks is already part of infrastructure for pretty wide amount
of plugins (Katello, Remote Execution, Chef, Salt, Ansible, Docker…),
it seems moving foreman-tasks code-base into the core as part of the
Foreman is where people want to get us moving.

Therefore, let me start proposing the plan for moving this subject forward.

The plan would be:

1. get the the foreman-tasks-core, that is part of the tasks code
   being use both in foreman server and smart-proxy code into separate
   repository
2. get the foreman-tasks on the same set of rubocop rules the foreman
   code uses
3. fill unit test gaps
4. open PR to foreman with the foreman-tasks functionality
5. open PR to get the hammer-cli-foreman-tasks inside hammer-cli-foreman

The target version for this would be foreman 1.15.

In the mean time, I would like to kindly ask for trying to
find an acceptable workaround in [2] and [3] until this is done.

Also, since this will be pretty large change already, I would like to
avoid radical changes done as part of the PR: as already said,
foreman-tasks has already quite significant user base though various
plugins that we need to support and I would rather prefer keeping
those as additional issues to track and address based on priorities,
as well as going though proper deprecation cycle if we need to
deprecate something.

Opinions, comments, questions, concerns (and remember, is also a
valid response:)

[1] - https://github.com/theforeman/foreman/pull/3670
[2] - https://github.com/theforeman/foreman-packaging/pull/1436
[3] - https://github.com/theforeman/foreman-packaging/pull/1437

– Ivan

[mmoll]

Hi,

> In the mean time, I would like to kindly ask for trying to
> find an acceptable workaround in [2] and [3] until this is done.
>
> […]
>
> [2] - https://github.com/theforeman/foreman-packaging/pull/1436
> [3] - https://github.com/theforeman/foreman-packaging/pull/1437

Regarding [3], IMHO this can get merged as a workaround for now, to
unblock builds, I'm running the resulting package in my nightly install
and it seems to work. After that (and the RPM part also is done),
https://github.com/theforeman/puppet-foreman/pull/504 can get merged,
which should also unblock the nightly BATS runs.

For DEB (I think that's not possible for RPM) I'd then start to work on
replacing the current foreman-tasks plugin package with the core one,
which will ease the transitioning packaging-wise when the foreman-tasks
code is merged into core.

Regards

···

On Wed, Jan 04, 2017 at 10:29:19AM +0100, Ivan Necas wrote: -- Michael Moll

[Marek_Hulan]

+1

···

-- Marek

On středa 4. ledna 2017 10:29:19 CET Ivan Necas wrote:

Hello friends,

You’ve might noticed we started using foreman-tasks inside the
foreman-core since [1] was merged. Unfortunately, we hit issues in the
packaging phase and the proposed workarounds were not accepted ([2]
[3]).

Based on various discussions, as well as on the fact that
foreman-tasks is already part of infrastructure for pretty wide amount
of plugins (Katello, Remote Execution, Chef, Salt, Ansible, Docker…),
it seems moving foreman-tasks code-base into the core as part of the
Foreman is where people want to get us moving.

Therefore, let me start proposing the plan for moving this subject forward.

The plan would be:

1. get the the foreman-tasks-core, that is part of the tasks code
   being use both in foreman server and smart-proxy code into separate
   repository
2. get the foreman-tasks on the same set of rubocop rules the foreman
   code uses
3. fill unit test gaps
4. open PR to foreman with the foreman-tasks functionality
5. open PR to get the hammer-cli-foreman-tasks inside hammer-cli-foreman

The target version for this would be foreman 1.15.

In the mean time, I would like to kindly ask for trying to
find an acceptable workaround in [2] and [3] until this is done.

Also, since this will be pretty large change already, I would like to
avoid radical changes done as part of the PR: as already said,
foreman-tasks has already quite significant user base though various
plugins that we need to support and I would rather prefer keeping
those as additional issues to track and address based on priorities,
as well as going though proper deprecation cycle if we need to
deprecate something.

Opinions, comments, questions, concerns (and remember, is also a
valid response:)

[1] - Fixes #15779 - make background processing available by ares · Pull Request #3670 · theforeman/foreman · GitHub
[2] - Refs #15779 - add dependency to foreman-tasks (rpm) by iNecas · Pull Request #1436 · theforeman/foreman-packaging · GitHub
[3] - Refs #15779 - handle foreman-tasks in core (DEB) by mmoll · Pull Request #1437 · theforeman/foreman-packaging · GitHub

– Ivan

[ehelms]

> Hello friends,
>
> You've might noticed we started using foreman-tasks inside the
> foreman-core since [1] was merged. Unfortunately, we hit issues in the
> packaging phase and the proposed workarounds were not accepted ([2]
> [3]).
>
> Based on various discussions, as well as on the fact that
> foreman-tasks is already part of infrastructure for pretty wide amount
> of plugins (Katello, Remote Execution, Chef, Salt, Ansible, Docker…),
> it seems moving foreman-tasks code-base into the core as part of the
> Foreman is where people want to get us moving.
>

+1 given that foreman-tasks has proven itself to be a corner stone of
tooling for many plugins including core now.

>
> Therefore, let me start proposing the plan for moving this subject forward.
>
> The plan would be:
>
> 1. get the the foreman-tasks-core, that is part of the tasks code
> being use both in foreman server and smart-proxy code into separate
> repository
> 2. get the foreman-tasks on the same set of rubocop rules the foreman
> code uses
> 3. fill unit test gaps
> 4. open PR to foreman with the foreman-tasks functionality
> 5. open PR to get the hammer-cli-foreman-tasks inside hammer-cli-foreman
>
> The target version for this would be foreman 1.15.
>

Beyond the version being targeted, do you have a time line for this? The
sooner the better for many reasons.

>
> In the mean time, I would like to kindly ask for trying to
> find an acceptable workaround in [2] and [3] until this is done.
>

I commented (sorry for not jumping in sooner).

>
> Also, since this will be pretty large change already, I would like to
> avoid radical changes done as part of the PR: as already said,
> foreman-tasks has already quite significant user base though various
> plugins that we need to support and I would rather prefer keeping
> those as additional issues to track and address based on priorities,
> as well as going though proper deprecation cycle if we need to
> deprecate something.
>

+1 – nightly is stagnating a bit for various plugins and core, unsticking
that even with a temporary work around would be great.

I'd even be OK with foreman-tasks going into the core repository as is as
an engine in a sub-directory. Similar to [1]. I realize this may not be
ideal long term, but it has the benefits of being able to migrate the code
quicker, and sooner. The code would now live in core as is (thus in theory
less prone to potential breakages) and could then be migrated out into some
more official code layout that developers desire.

[1] katello/engines/bastion_katello at master · Katello/katello · GitHub

···

On Wed, Jan 4, 2017 at 4:29 AM, Ivan Necas wrote:

Opinions, comments, questions, concerns (and remember, is also a
valid response:)

[1] - Fixes #15779 - make background processing available by ares · Pull Request #3670 · theforeman/foreman · GitHub
[2] - Refs #15779 - add dependency to foreman-tasks (rpm) by iNecas · Pull Request #1436 · theforeman/foreman-packaging · GitHub
[3] - Refs #15779 - handle foreman-tasks in core (DEB) by mmoll · Pull Request #1437 · theforeman/foreman-packaging · GitHub

– Ivan

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering

[lzap]

Is it only the Rails App that depends on Foreman Core? What's reasonable
plan to me is:

a) Extract core part that does not depend on Foreman core into a simple gem
(that's what you propose I guess) and make it a hard dependency of core.
b) The rest of foreman-tasks (Rails App/Engine - administrative UI part for
tasks) remains an optional plugin in a separate (plugin) repository.
c) We include the plugin in the default installation set but Foreman should
work without the plugin as well, it is only needed to perform
administrative tasks.

If that's what you propose, why do we need temporary workarounds? Can't we
just do it right away? The timing is good, 1.14 is getting out soon, we can
start digging.

···

On Wed, Jan 4, 2017 at 10:29 AM, Ivan Necas wrote:

Hello friends,

You’ve might noticed we started using foreman-tasks inside the
foreman-core since [1] was merged. Unfortunately, we hit issues in the
packaging phase and the proposed workarounds were not accepted ([2]
[3]).

Based on various discussions, as well as on the fact that
foreman-tasks is already part of infrastructure for pretty wide amount
of plugins (Katello, Remote Execution, Chef, Salt, Ansible, Docker…),
it seems moving foreman-tasks code-base into the core as part of the
Foreman is where people want to get us moving.

Therefore, let me start proposing the plan for moving this subject forward.

The plan would be:

1. get the the foreman-tasks-core, that is part of the tasks code
   being use both in foreman server and smart-proxy code into separate
   repository
2. get the foreman-tasks on the same set of rubocop rules the foreman
   code uses
3. fill unit test gaps
4. open PR to foreman with the foreman-tasks functionality
5. open PR to get the hammer-cli-foreman-tasks inside hammer-cli-foreman

The target version for this would be foreman 1.15.

In the mean time, I would like to kindly ask for trying to
find an acceptable workaround in [2] and [3] until this is done.

Also, since this will be pretty large change already, I would like to
avoid radical changes done as part of the PR: as already said,
foreman-tasks has already quite significant user base though various
plugins that we need to support and I would rather prefer keeping
those as additional issues to track and address based on priorities,
as well as going though proper deprecation cycle if we need to
deprecate something.

Opinions, comments, questions, concerns (and remember, is also a
valid response:)

[1] - Fixes #15779 - make background processing available by ares · Pull Request #3670 · theforeman/foreman · GitHub
[2] - Refs #15779 - add dependency to foreman-tasks (rpm) by iNecas · Pull Request #1436 · theforeman/foreman-packaging · GitHub
[3] - Refs #15779 - handle foreman-tasks in core (DEB) by mmoll · Pull Request #1437 · theforeman/foreman-packaging · GitHub

– Ivan

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas @lzap Zapletal

[Marek_Hulan]

Few comments below

> Is it only the Rails App that depends on Foreman Core? What's reasonable
> plan to me is:
>
> a) Extract core part that does not depend on Foreman core into a simple gem
> (that's what you propose I guess) and make it a hard dependency of core.
> b) The rest of foreman-tasks (Rails App/Engine - administrative UI part for
> tasks) remains an optional plugin in a separate (plugin) repository.
> c) We include the plugin in the default installation set but Foreman should
> work without the plugin as well, it is only needed to perform
> administrative tasks.

I'm not sure I follow what you mean by administrative tasks. Note that reports
import and puppet envs import are core actiones that now run as a foreman task
(both synchronous and asynchronous variants). By making the UI part optional
users would not be able to monitor their progress, cancel them etc if they
don't install the plugin. What would be the benefit of such setup?

> If that's what you propose, why do we need temporary workarounds? Can't we
> just do it right away? The timing is good, 1.14 is getting out soon, we can
> start digging.

Moving the code into core code base might take time until it's reviewed and
merged. The workaround would get nightly builds back before the code is moved.

···

On čtvrtek 5. ledna 2017 15:25:47 CET Lukas Zapletal wrote:

–
Marek

On Wed, Jan 4, 2017 at 10:29 AM, Ivan Necas inecas@redhat.com wrote:

Hello friends,

You’ve might noticed we started using foreman-tasks inside the
foreman-core since [1] was merged. Unfortunately, we hit issues in the
packaging phase and the proposed workarounds were not accepted ([2]
[3]).

Based on various discussions, as well as on the fact that
foreman-tasks is already part of infrastructure for pretty wide amount
of plugins (Katello, Remote Execution, Chef, Salt, Ansible, Docker…),
it seems moving foreman-tasks code-base into the core as part of the
Foreman is where people want to get us moving.

Therefore, let me start proposing the plan for moving this subject
forward.

The plan would be:

1. get the the foreman-tasks-core, that is part of the tasks code
   being use both in foreman server and smart-proxy code into separate
   repository
2. get the foreman-tasks on the same set of rubocop rules the foreman
   code uses
3. fill unit test gaps
4. open PR to foreman with the foreman-tasks functionality
5. open PR to get the hammer-cli-foreman-tasks inside hammer-cli-foreman

The target version for this would be foreman 1.15.

In the mean time, I would like to kindly ask for trying to
find an acceptable workaround in [2] and [3] until this is done.

Also, since this will be pretty large change already, I would like to
avoid radical changes done as part of the PR: as already said,
foreman-tasks has already quite significant user base though various
plugins that we need to support and I would rather prefer keeping
those as additional issues to track and address based on priorities,
as well as going though proper deprecation cycle if we need to
deprecate something.

Opinions, comments, questions, concerns (and remember, is also a
valid response:)

[1] - Fixes #15779 - make background processing available by ares · Pull Request #3670 · theforeman/foreman · GitHub
[2] - Refs #15779 - add dependency to foreman-tasks (rpm) by iNecas · Pull Request #1436 · theforeman/foreman-packaging · GitHub
[3] - Refs #15779 - handle foreman-tasks in core (DEB) by mmoll · Pull Request #1437 · theforeman/foreman-packaging · GitHub

– Ivan

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lzap]

>
> I'm not sure I follow what you mean by administrative tasks. Note that
> reports
> import and puppet envs import are core actiones that now run as a foreman
> task
> (both synchronous and asynchronous variants). By making the UI part
> optional
> users would not be able to monitor their progress, cancel them etc if they
> don't install the plugin. What would be the benefit of such setup?
>

I am not telling not to ship it, but making it optional (but installed by
default). Assuming that processes would work normally without the UI part.

Other option is to simply move the UI into core, I don't think we should
make our decomposition effort a dogma. Let's be realistic.

Benefit? Solving the stalemate perhaps

> Moving the code into core code base might take time until it's reviewed and
> merged. The workaround would get nightly builds back before the code is
> moved.
>

Oh that Sure.

[iNecas]

Lukas Zapletal <lzap@redhat.com> writes:

>>
>> I'm not sure I follow what you mean by administrative tasks. Note that
>> reports
>> import and puppet envs import are core actiones that now run as a foreman
>> task
>> (both synchronous and asynchronous variants). By making the UI part
>> optional
>> users would not be able to monitor their progress, cancel them etc if they
>> don't install the plugin. What would be the benefit of such setup?
>>
>
> I am not telling not to ship it, but making it optional (but installed by
> default). Assuming that processes would work normally without the UI part.
>
> Other option is to simply move the UI into core, I don't think we should
> make our decomposition effort a dogma. Let's be realistic.
>
> Benefit? Solving the stalemate perhaps

So the latest updates updates if you don't follow the packaging PR [1]

The goal right now is to:

1. unblock the nightlies
2. keep the async operations possible
3. prefer proper way over hacks

Although the goal is to get the foreman-tasks functionality to the core,
I don't think we can effort blocking nightlies much longer. To preserve
the async possibility there.

I'm looking into leveraging ActiveJob interface to define the async
operations we added. The idea is: when there is no foreman-tasks, the
in-thread executor, that already is build into Rails, will be used, and
from the end user, it will behave as before.

However, when foreman-tasks will be around and configured to be used for
async processing, the operation will go though that.

Once this would be done, we could start adding the UI around async
operations directly to the core: so the foreman-tasks functionality
would be gradually moved to the core this way: once the core is given certain
functionality, it can be removed from the tasks.

The important thing here is we could work on enhancing tasking
infrastructure in core while still supporting the current foreman-tasks
users and delivering enhancements there in the meantime.

I'm setting the deadline for this on Monday EOB. If by that time turns
our this plan is not feasible, we would need to sacrifice goal 1., 2. or
3.

– Ivan

[1] https://github.com/theforeman/foreman-packaging/pull/1436#issuecomment-273780029

···

> > >> Moving the code into core code base might take time until it's reviewed and >> merged. The workaround would get nightly builds back before the code is >> moved. >> > > Oh that :-D Sure. > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

[ekohl]

To me this sounds like a good plan.

···

On Thu, Jan 19, 2017 at 06:04:58PM +0100, Ivan Necas wrote: > Lukas Zapletal writes: > > >> > >> I'm not sure I follow what you mean by administrative tasks. Note that > >> reports > >> import and puppet envs import are core actiones that now run as a foreman > >> task > >> (both synchronous and asynchronous variants). By making the UI part > >> optional > >> users would not be able to monitor their progress, cancel them etc if they > >> don't install the plugin. What would be the benefit of such setup? > >> > > > > I am not telling not to ship it, but making it optional (but installed by > > default). Assuming that processes would work normally without the UI part. > > > > Other option is to simply move the UI into core, I don't think we should > > make our decomposition effort a dogma. Let's be realistic. > > > > Benefit? Solving the stalemate perhaps :-) > > So the latest updates updates if you don't follow the packaging PR [1] > > The goal right now is to: > > 1. unblock the nightlies > 2. keep the async operations possible > 3. prefer proper way over hacks > > Although the goal is to get the foreman-tasks functionality to the core, > I don't think we can effort blocking nightlies much longer. To preserve > the async possibility there. > > I'm looking into leveraging ActiveJob interface to define the async > operations we added. The idea is: when there is no foreman-tasks, the > in-thread executor, that already is build into Rails, will be used, and > from the end user, it will behave as before. > > However, when foreman-tasks will be around and configured to be used for > async processing, the operation will go though that. > > Once this would be done, we could start adding the UI around async > operations directly to the core: so the foreman-tasks functionality > would be gradually moved to the core this way: once the core is given certain > functionality, it can be removed from the tasks. > > The important thing here is we could work on enhancing tasking > infrastructure in core while still supporting the current foreman-tasks > users and delivering enhancements there in the meantime. > > I'm setting the deadline for this on Monday EOB. If by that time turns > our this plan is not feasible, we would need to sacrifice goal 1., 2. or > 3.

[Marek_Hulan]

+1

If active job does not work we could also fallback to something like
if defined?(ForemanTask) and keep working on active job alternative.

···

On pátek 20. ledna 2017 2:51:06 CET Ewoud Kohl van Wijngaarden wrote: > On Thu, Jan 19, 2017 at 06:04:58PM +0100, Ivan Necas wrote: > > Lukas Zapletal writes: > > >> I'm not sure I follow what you mean by administrative tasks. Note that > > >> reports > > >> import and puppet envs import are core actiones that now run as a > > >> foreman > > >> task > > >> (both synchronous and asynchronous variants). By making the UI part > > >> optional > > >> users would not be able to monitor their progress, cancel them etc if > > >> they > > >> don't install the plugin. What would be the benefit of such setup? > > > > > > I am not telling not to ship it, but making it optional (but installed > > > by > > > default). Assuming that processes would work normally without the UI > > > part. > > > > > > Other option is to simply move the UI into core, I don't think we should > > > make our decomposition effort a dogma. Let's be realistic. > > > > > > Benefit? Solving the stalemate perhaps :-) > > > > So the latest updates updates if you don't follow the packaging PR [1] > > > > The goal right now is to: > > 1. unblock the nightlies > > 2. keep the async operations possible > > 3. prefer proper way over hacks > > > > Although the goal is to get the foreman-tasks functionality to the core, > > I don't think we can effort blocking nightlies much longer. To preserve > > the async possibility there. > > > > I'm looking into leveraging ActiveJob interface to define the async > > operations we added. The idea is: when there is no foreman-tasks, the > > in-thread executor, that already is build into Rails, will be used, and > > from the end user, it will behave as before. > > > > However, when foreman-tasks will be around and configured to be used for > > async processing, the operation will go though that. > > > > Once this would be done, we could start adding the UI around async > > operations directly to the core: so the foreman-tasks functionality > > would be gradually moved to the core this way: once the core is given > > certain functionality, it can be removed from the tasks. > > > > The important thing here is we could work on enhancing tasking > > infrastructure in core while still supporting the current foreman-tasks > > users and delivering enhancements there in the meantime. > > > > I'm setting the deadline for this on Monday EOB. If by that time turns > > our this plan is not feasible, we would need to sacrifice goal 1., 2. or > > 3. > > To me this sounds like a good plan.

–
Marek

[lzap]

Why we haven't considered ActiveJob API in the first place? This looks
like ideal solution, easy development and documented API.

Can't ActiveJob help us with memory issues we have with external task
executor in Katello? I mean what if we run background tasks inside
passenger process by default for easier deployment and maintenance. Of
course, restart would mean lost queue.

LZ

···

On Fri, Jan 20, 2017 at 9:26 AM, Marek Hulán wrote: > On pátek 20. ledna 2017 2:51:06 CET Ewoud Kohl van Wijngaarden wrote: >> On Thu, Jan 19, 2017 at 06:04:58PM +0100, Ivan Necas wrote: >> > Lukas Zapletal writes: >> > >> I'm not sure I follow what you mean by administrative tasks. Note that >> > >> reports >> > >> import and puppet envs import are core actiones that now run as a >> > >> foreman >> > >> task >> > >> (both synchronous and asynchronous variants). By making the UI part >> > >> optional >> > >> users would not be able to monitor their progress, cancel them etc if >> > >> they >> > >> don't install the plugin. What would be the benefit of such setup? >> > > >> > > I am not telling not to ship it, but making it optional (but installed >> > > by >> > > default). Assuming that processes would work normally without the UI >> > > part. >> > > >> > > Other option is to simply move the UI into core, I don't think we should >> > > make our decomposition effort a dogma. Let's be realistic. >> > > >> > > Benefit? Solving the stalemate perhaps :-) >> > >> > So the latest updates updates if you don't follow the packaging PR [1] >> > >> > The goal right now is to: >> > 1. unblock the nightlies >> > 2. keep the async operations possible >> > 3. prefer proper way over hacks >> > >> > Although the goal is to get the foreman-tasks functionality to the core, >> > I don't think we can effort blocking nightlies much longer. To preserve >> > the async possibility there. >> > >> > I'm looking into leveraging ActiveJob interface to define the async >> > operations we added. The idea is: when there is no foreman-tasks, the >> > in-thread executor, that already is build into Rails, will be used, and >> > from the end user, it will behave as before. >> > >> > However, when foreman-tasks will be around and configured to be used for >> > async processing, the operation will go though that. >> > >> > Once this would be done, we could start adding the UI around async >> > operations directly to the core: so the foreman-tasks functionality >> > would be gradually moved to the core this way: once the core is given >> > certain functionality, it can be removed from the tasks. >> > >> > The important thing here is we could work on enhancing tasking >> > infrastructure in core while still supporting the current foreman-tasks >> > users and delivering enhancements there in the meantime. >> > >> > I'm setting the deadline for this on Monday EOB. If by that time turns >> > our this plan is not feasible, we would need to sacrifice goal 1., 2. or >> > 3. >> >> To me this sounds like a good plan. > > +1 > > If active job does not work we could also fallback to something like > `if defined?(ForemanTask)` and keep working on active job alternative. > > -- > Marek > > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas @lzap Zapletal

[iNecas]

Lukas Zapletal <lzap@redhat.com> writes:

> Why we haven't considered ActiveJob API in the first place? This looks
> like ideal solution, easy development and documented API.
>
> Can't ActiveJob help us with memory issues we have with external task
> executor in Katello? I mean what if we run background tasks inside
> passenger process by default for easier deployment and maintenance. Of
> course, restart would mean lost queue.

You immediately described what the issue with running things inside
passenger is: and as we orchestrate tasks though multiple services and
though a lot of API calls and sometimes though a significant time, this
is not really a solution for this kind of operations.

If Katello needed just an async processing, we would not need to put our
effort into Dynflow in the first place. But Katello needs orchestration,
(similarly as Host creation does and it's one of the reasons we try to
incorporate tasks inside core is ability to leverage this for host
creation once). ActiveJob is just a simple API about single async
operation: it's where it fits for the current usage of async processing
we have in Foreman but trying to use it for more that that would be a
mistake IMO.

To be cintroducing ActiveJob API doesn't mean deprecation of Dynflow.
What introduction of ActiveJob (besides the async processing itself)
brings us is an opportunity to also work more around the tasks UI in
general, that will be usable by both generic Acitve jobs, as well as
orchestrations run by Dynflow.

BTW I believe there is way how to refactor Dynflow to actually not
need a separate executor and process the queue inside Passenger
processes. But I would need few free weekends for a POC on this

– Ivan

···

> > LZ > > On Fri, Jan 20, 2017 at 9:26 AM, Marek Hulán wrote: >> On pátek 20. ledna 2017 2:51:06 CET Ewoud Kohl van Wijngaarden wrote: >>> On Thu, Jan 19, 2017 at 06:04:58PM +0100, Ivan Necas wrote: >>> > Lukas Zapletal writes: >>> > >> I'm not sure I follow what you mean by administrative tasks. Note that >>> > >> reports >>> > >> import and puppet envs import are core actiones that now run as a >>> > >> foreman >>> > >> task >>> > >> (both synchronous and asynchronous variants). By making the UI part >>> > >> optional >>> > >> users would not be able to monitor their progress, cancel them etc if >>> > >> they >>> > >> don't install the plugin. What would be the benefit of such setup? >>> > > >>> > > I am not telling not to ship it, but making it optional (but installed >>> > > by >>> > > default). Assuming that processes would work normally without the UI >>> > > part. >>> > > >>> > > Other option is to simply move the UI into core, I don't think we should >>> > > make our decomposition effort a dogma. Let's be realistic. >>> > > >>> > > Benefit? Solving the stalemate perhaps :-) >>> > >>> > So the latest updates updates if you don't follow the packaging PR [1] >>> > >>> > The goal right now is to: >>> > 1. unblock the nightlies >>> > 2. keep the async operations possible >>> > 3. prefer proper way over hacks >>> > >>> > Although the goal is to get the foreman-tasks functionality to the core, >>> > I don't think we can effort blocking nightlies much longer. To preserve >>> > the async possibility there. >>> > >>> > I'm looking into leveraging ActiveJob interface to define the async >>> > operations we added. The idea is: when there is no foreman-tasks, the >>> > in-thread executor, that already is build into Rails, will be used, and >>> > from the end user, it will behave as before. >>> > >>> > However, when foreman-tasks will be around and configured to be used for >>> > async processing, the operation will go though that. >>> > >>> > Once this would be done, we could start adding the UI around async >>> > operations directly to the core: so the foreman-tasks functionality >>> > would be gradually moved to the core this way: once the core is given >>> > certain functionality, it can be removed from the tasks. >>> > >>> > The important thing here is we could work on enhancing tasking >>> > infrastructure in core while still supporting the current foreman-tasks >>> > users and delivering enhancements there in the meantime. >>> > >>> > I'm setting the deadline for this on Monday EOB. If by that time turns >>> > our this plan is not feasible, we would need to sacrifice goal 1., 2. or >>> > 3. >>> >>> To me this sounds like a good plan. >> >> +1 >> >> If active job does not work we could also fallback to something like >> `if defined?(ForemanTask)` and keep working on active job alternative. >> >> -- >> Marek >> >> >> -- >> You received this message because you are subscribed to the Google Groups "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > > -- > Later, > Lukas @lzap Zapletal > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

[ohadlevy]

> Why we haven't considered ActiveJob API in the first place? This looks
> like ideal solution, easy development and documented API.
>
> Can't ActiveJob help us with memory issues we have with external task
> executor in Katello? I mean what if we run background tasks inside
> passenger process by default for easier deployment and maintenance. Of
> course, restart would mean lost queue.
>

I don't know if that's what you had in mind or not, but if it end up being
blocking the request, its not useful at all, as that would lead to more
confusion, and a developer will need to care for more usage cases (is it
blocking now, is it running async) and especially when there is a need for
ui, imho it just makes it harder.

if we do move to active job api in core(+1), I could see us starting
without tasks first for pure async operations, over time converting dynflow
to be orchestration engine vs async and orchestration.
Futher, I would challenge a bit the orchestration implementation in Katello
(not in a negative way) but if a task fails due to unavailability of one of
the backends, the application should know how to deal with it (micro
service / soa) rather then fail / lock the task for the future, I would ask
if this is the correct long term approach for complex operations.

Ohad

···

On Fri, Jan 20, 2017 at 7:14 PM, Lukas Zapletal wrote:

LZ

On Fri, Jan 20, 2017 at 9:26 AM, Marek Hulán mhulan@redhat.com wrote:

On pátek 20. ledna 2017 2:51:06 CET Ewoud Kohl van Wijngaarden wrote:

On Thu, Jan 19, 2017 at 06:04:58PM +0100, Ivan Necas wrote:

Lukas Zapletal lzap@redhat.com writes:

I’m not sure I follow what you mean by administrative tasks. Note
that
reports
import and puppet envs import are core actiones that now run as a
foreman
task
(both synchronous and asynchronous variants). By making the UI part
optional
users would not be able to monitor their progress, cancel them etc
if
they
don’t install the plugin. What would be the benefit of such setup?

I am not telling not to ship it, but making it optional (but
installed
by
default). Assuming that processes would work normally without the UI
part.

Other option is to simply move the UI into core, I don’t think we
should
make our decomposition effort a dogma. Let’s be realistic.

Benefit? Solving the stalemate perhaps

So the latest updates updates if you don’t follow the packaging PR [1]

The goal right now is to:

1. unblock the nightlies
2. keep the async operations possible
3. prefer proper way over hacks

Although the goal is to get the foreman-tasks functionality to the
core,
I don’t think we can effort blocking nightlies much longer. To
preserve
the async possibility there.

I’m looking into leveraging ActiveJob interface to define the async
operations we added. The idea is: when there is no foreman-tasks, the
in-thread executor, that already is build into Rails, will be used,
and
from the end user, it will behave as before.

However, when foreman-tasks will be around and configured to be used
for
async processing, the operation will go though that.

Once this would be done, we could start adding the UI around async
operations directly to the core: so the foreman-tasks functionality
would be gradually moved to the core this way: once the core is given
certain functionality, it can be removed from the tasks.

The important thing here is we could work on enhancing tasking
infrastructure in core while still supporting the current
foreman-tasks
users and delivering enhancements there in the meantime.

I’m setting the deadline for this on Monday EOB. If by that time turns
our this plan is not feasible, we would need to sacrifice goal 1., 2.
or
3.

To me this sounds like a good plan.

+1

If active job does not work we could also fallback to something like
if defined?(ForemanTask) and keep working on active job alternative.

–
Marek

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas @lzap Zapletal

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[iNecas]

Ohad Levy <ohadlevy@gmail.com> writes:

>
>> Why we haven't considered ActiveJob API in the first place? This looks
>> like ideal solution, easy development and documented API.
>>
>> Can't ActiveJob help us with memory issues we have with external task
>> executor in Katello? I mean what if we run background tasks inside
>> passenger process by default for easier deployment and maintenance. Of
>> course, restart would mean lost queue.
>>
>
> I don't know if that's what you had in mind or not, but if it end up being
> blocking the request, its not useful at all, as that would lead to more
> confusion, and a developer will need to care for more usage cases (is it
> blocking now, is it running async) and especially when there is a need for
> ui, imho it just makes it harder.
>
> if we do move to active job api in core(+1), I could see us starting
> without tasks first for pure async operations, over time converting dynflow
> to be orchestration engine vs async and orchestration.
> Futher, I would challenge a bit the orchestration implementation in Katello
> (not in a negative way) but if a task fails due to unavailability of one of
> the backends, the application should know how to deal with it (micro
> service / soa) rather then fail / lock the task for the future, I would ask
> if this is the correct long term approach for complex operations.

While I like challenging the things, let's not make this thread to
broad.

I've made quite a progress with ActiveJob, and while it's not perfect,
for basic introduction of async operations it should work well enough.

However, I'm still few days from having it ready, and I want keep the
promise to propose the solution to falling nightlies by Monday evening.
Therefore, I've opened a PR for reverting the original change

Refs #15779 - make background processing unavailable for now by iNecas · Pull Request #4217 · theforeman/foreman · GitHub

This is not a resignation of trying to get the tasks functionality into
core, but rather lowering a pressure a bit so that we can focus on the
right things to do. I hope the PR adding the functionality back, using
the approach we talked about, will come sooner rather than later, followed
by adding the tasking plumbing into core, to support async operations,
as well as dynflow orchestration and current host orchestration.

– Ivan

···

> On Fri, Jan 20, 2017 at 7:14 PM, Lukas Zapletal wrote:

Ohad

LZ

On Fri, Jan 20, 2017 at 9:26 AM, Marek Hulán mhulan@redhat.com wrote:

On pátek 20. ledna 2017 2:51:06 CET Ewoud Kohl van Wijngaarden wrote:

On Thu, Jan 19, 2017 at 06:04:58PM +0100, Ivan Necas wrote:

Lukas Zapletal lzap@redhat.com writes:

I’m not sure I follow what you mean by administrative tasks. Note
that
reports
import and puppet envs import are core actiones that now run as a
foreman
task
(both synchronous and asynchronous variants). By making the UI part
optional
users would not be able to monitor their progress, cancel them etc
if
they
don’t install the plugin. What would be the benefit of such setup?

I am not telling not to ship it, but making it optional (but
installed
by
default). Assuming that processes would work normally without the UI
part.

Other option is to simply move the UI into core, I don’t think we
should
make our decomposition effort a dogma. Let’s be realistic.

Benefit? Solving the stalemate perhaps

So the latest updates updates if you don’t follow the packaging PR [1]

The goal right now is to:

1. unblock the nightlies
2. keep the async operations possible
3. prefer proper way over hacks

Although the goal is to get the foreman-tasks functionality to the
core,
I don’t think we can effort blocking nightlies much longer. To
preserve
the async possibility there.

I’m looking into leveraging ActiveJob interface to define the async
operations we added. The idea is: when there is no foreman-tasks, the
in-thread executor, that already is build into Rails, will be used,
and
from the end user, it will behave as before.

However, when foreman-tasks will be around and configured to be used
for
async processing, the operation will go though that.

Once this would be done, we could start adding the UI around async
operations directly to the core: so the foreman-tasks functionality
would be gradually moved to the core this way: once the core is given
certain functionality, it can be removed from the tasks.

The important thing here is we could work on enhancing tasking
infrastructure in core while still supporting the current
foreman-tasks
users and delivering enhancements there in the meantime.

I’m setting the deadline for this on Monday EOB. If by that time turns
our this plan is not feasible, we would need to sacrifice goal 1., 2.
or
3.

To me this sounds like a good plan.

+1

If active job does not work we could also fallback to something like
if defined?(ForemanTask) and keep working on active job alternative.

–
Marek

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas @lzap Zapletal

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ekohl]

I greatly appreciate your effort to fix the situation.

Given nightly has already been broken for a bit, I'd find it acceptable
if we pushed the reversal to next week. That should give you a little
bit extra time to work on ActiveJob which is most likely where we want
to end up anyway.

My biggest question is: in a few days (I'm guessing that's 3 or 4), will
it be in mergeable shape? Would that mean we can expect it to be merged
on Monday? With FOSDEM and cfgmgtcamp coming up I'd like to have nightly
back into shape before it.

I'd propose to aim for working nightlies built on Wednesday 1 February.
We most likely want a day to fix any regressions that might have slipped
in so we need to merge something to Foreman develop on Monday 30
January. Since reviewers need time to review that means the ActiveJob PR
needs to be there some time in advance. I can't speak for the reviewers,
but I'm guessing before Monday morning would be nice.

If no consensus on the ActiveJob PR can be reached by Monday evening,
then the reversal is merged instead. This still allows to verify/fix
other regressions on Tuesday and have working nightlies on Wednesday.

···

On Tue, Jan 24, 2017 at 12:10:34AM +0100, Ivan Necas wrote: > Ohad Levy writes: > > > On Fri, Jan 20, 2017 at 7:14 PM, Lukas Zapletal wrote: > > > >> Why we haven't considered ActiveJob API in the first place? This looks > >> like ideal solution, easy development and documented API. > >> > >> Can't ActiveJob help us with memory issues we have with external task > >> executor in Katello? I mean what if we run background tasks inside > >> passenger process by default for easier deployment and maintenance. Of > >> course, restart would mean lost queue. > >> > > > > I don't know if that's what you had in mind or not, but if it end up being > > blocking the request, its not useful at all, as that would lead to more > > confusion, and a developer will need to care for more usage cases (is it > > blocking now, is it running async) and especially when there is a need for > > ui, imho it just makes it harder. > > > > if we do move to active job api in core(+1), I could see us starting > > without tasks first for pure async operations, over time converting dynflow > > to be orchestration engine vs async and orchestration. > > Futher, I would challenge a bit the orchestration implementation in Katello > > (not in a negative way) but if a task fails due to unavailability of one of > > the backends, the application should know how to deal with it (micro > > service / soa) rather then fail / lock the task for the future, I would ask > > if this is the correct long term approach for complex operations. > > While I like challenging the things, let's not make this thread to > broad. > > I've made quite a progress with ActiveJob, and while it's not perfect, > for basic introduction of async operations it should work well enough. > > However, I'm still few days from having it ready, and I want keep the > promise to propose the solution to falling nightlies by Monday evening. > Therefore, I've opened a PR for reverting the original change > > https://github.com/theforeman/foreman/pull/4217 > > This is not a resignation of trying to get the tasks functionality into > core, but rather lowering a pressure a bit so that we can focus on the > right things to do. I hope the PR adding the functionality back, using > the approach we talked about, will come sooner rather than later, followed > by adding the tasking plumbing into core, to support async operations, > as well as dynflow orchestration and current host orchestration.

[iNecas]

Ewoud Kohl van Wijngaarden <ewoud@kohlvanwijngaarden.nl> writes:

>> Ohad Levy <ohadlevy@gmail.com> writes:
>>
>> >
>> >> Why we haven't considered ActiveJob API in the first place? This looks
>> >> like ideal solution, easy development and documented API.
>> >>
>> >> Can't ActiveJob help us with memory issues we have with external task
>> >> executor in Katello? I mean what if we run background tasks inside
>> >> passenger process by default for easier deployment and maintenance. Of
>> >> course, restart would mean lost queue.
>> >>
>> >
>> > I don't know if that's what you had in mind or not, but if it end up being
>> > blocking the request, its not useful at all, as that would lead to more
>> > confusion, and a developer will need to care for more usage cases (is it
>> > blocking now, is it running async) and especially when there is a need for
>> > ui, imho it just makes it harder.
>> >
>> > if we do move to active job api in core(+1), I could see us starting
>> > without tasks first for pure async operations, over time converting dynflow
>> > to be orchestration engine vs async and orchestration.
>> > Futher, I would challenge a bit the orchestration implementation in Katello
>> > (not in a negative way) but if a task fails due to unavailability of one of
>> > the backends, the application should know how to deal with it (micro
>> > service / soa) rather then fail / lock the task for the future, I would ask
>> > if this is the correct long term approach for complex operations.
>>
>> While I like challenging the things, let's not make this thread to
>> broad.
>>
>> I've made quite a progress with ActiveJob, and while it's not perfect,
>> for basic introduction of async operations it should work well enough.
>>
>> However, I'm still few days from having it ready, and I want keep the
>> promise to propose the solution to falling nightlies by Monday evening.
>> Therefore, I've opened a PR for reverting the original change
>>
>> Refs #15779 - make background processing unavailable for now by iNecas · Pull Request #4217 · theforeman/foreman · GitHub
>>
>> This is not a resignation of trying to get the tasks functionality into
>> core, but rather lowering a pressure a bit so that we can focus on the
>> right things to do. I hope the PR adding the functionality back, using
>> the approach we talked about, will come sooner rather than later, followed
>> by adding the tasking plumbing into core, to support async operations,
>> as well as dynflow orchestration and current host orchestration.
>
> I greatly appreciate your effort to fix the situation.
>
> Given nightly has already been broken for a bit, I'd find it acceptable
> if we pushed the reversal to next week. That should give you a little
> bit extra time to work on ActiveJob which is most likely where we want
> to end up anyway.
>
> My biggest question is: in a few days (I'm guessing that's 3 or 4), will
> it be in mergeable shape? Would that mean we can expect it to be merged
> on Monday? With FOSDEM and cfgmgtcamp coming up I'd like to have nightly
> back into shape before it.

I think it's realistic. If it's acceptable for people to wait for few
days, I'm ok with the updated plan.

– Ivan

···

> On Tue, Jan 24, 2017 at 12:10:34AM +0100, Ivan Necas wrote: >> > On Fri, Jan 20, 2017 at 7:14 PM, Lukas Zapletal wrote:

I’d propose to aim for working nightlies built on Wednesday 1 February.
We most likely want a day to fix any regressions that might have slipped
in so we need to merge something to Foreman develop on Monday 30
January. Since reviewers need time to review that means the ActiveJob PR
needs to be there some time in advance. I can’t speak for the reviewers,
but I’m guessing before Monday morning would be nice.

If no consensus on the ActiveJob PR can be reached by Monday evening,
then the reversal is merged instead. This still allows to verify/fix
other regressions on Tuesday and have working nightlies on Wednesday.

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[ohadlevy]

> Ewoud Kohl van Wijngaarden <ewoud@kohlvanwijngaarden.nl> writes:
>
> >> Ohad Levy <ohadlevy@gmail.com> writes:
> >>
> >> >
> >> >> Why we haven't considered ActiveJob API in the first place? This
> looks
> >> >> like ideal solution, easy development and documented API.
> >> >>
> >> >> Can't ActiveJob help us with memory issues we have with external task
> >> >> executor in Katello? I mean what if we run background tasks inside
> >> >> passenger process by default for easier deployment and maintenance.
> Of
> >> >> course, restart would mean lost queue.
> >> >>
> >> >
> >> > I don't know if that's what you had in mind or not, but if it end up
> being
> >> > blocking the request, its not useful at all, as that would lead to
> more
> >> > confusion, and a developer will need to care for more usage cases (is
> it
> >> > blocking now, is it running async) and especially when there is a
> need for
> >> > ui, imho it just makes it harder.
> >> >
> >> > if we do move to active job api in core(+1), I could see us starting
> >> > without tasks first for pure async operations, over time converting
> dynflow
> >> > to be orchestration engine vs async and orchestration.
> >> > Futher, I would challenge a bit the orchestration implementation in
> Katello
> >> > (not in a negative way) but if a task fails due to unavailability of
> one of
> >> > the backends, the application should know how to deal with it (micro
> >> > service / soa) rather then fail / lock the task for the future, I
> would ask
> >> > if this is the correct long term approach for complex operations.
> >>
> >> While I like challenging the things, let's not make this thread to
> >> broad.
> >>
> >> I've made quite a progress with ActiveJob, and while it's not perfect,
> >> for basic introduction of async operations it should work well enough.
> >>
> >> However, I'm still few days from having it ready, and I want keep the
> >> promise to propose the solution to falling nightlies by Monday evening.
> >> Therefore, I've opened a PR for reverting the original change
> >>
> >> Refs #15779 - make background processing unavailable for now by iNecas · Pull Request #4217 · theforeman/foreman · GitHub
> >>
> >> This is not a resignation of trying to get the tasks functionality into
> >> core, but rather lowering a pressure a bit so that we can focus on the
> >> right things to do. I hope the PR adding the functionality back, using
> >> the approach we talked about, will come sooner rather than later,
> followed
> >> by adding the tasking plumbing into core, to support async operations,
> >> as well as dynflow orchestration and current host orchestration.
> >
> > I greatly appreciate your effort to fix the situation.
> >
> > Given nightly has already been broken for a bit, I'd find it acceptable
> > if we pushed the reversal to next week. That should give you a little
> > bit extra time to work on ActiveJob which is most likely where we want
> > to end up anyway.
> >
> > My biggest question is: in a few days (I'm guessing that's 3 or 4), will
> > it be in mergeable shape? Would that mean we can expect it to be merged
> > on Monday? With FOSDEM and cfgmgtcamp coming up I'd like to have nightly
> > back into shape before it.
>
> I think it's realistic. If it's acceptable for people to wait for few
> days, I'm ok with the updated plan.
>

TBH I dont fully understand the solution.

we will not ship foreman-tasks by default, but rather allow a developer to
write in a foreman-task compatible way (e.g. can work with and without
foreman tasks), and merge foreman tasks into core at some time in the
future.

What I don't get is:

1. if there is no foreman tasks, then regardless of how i execute (e.g. via
   active job) it would still be blocking, UI would need to account for that -
   meaning that we still going to have to modify the code that uses it.
2. we are going to remove foreman-tasks from core for the time being
   anyway, so why wait on the revert? tbh, report importing in the background
   is really low prio, and puppet class importing while useful, we can live
   without just as we did for 6 years or so before.

so unless I miss understand something, I would vote to remove it now to get
nightly back.

Ohad

···

On Tue, Jan 24, 2017 at 9:51 AM, Ivan Necas wrote: > > On Tue, Jan 24, 2017 at 12:10:34AM +0100, Ivan Necas wrote: > >> > On Fri, Jan 20, 2017 at 7:14 PM, Lukas Zapletal > wrote:

– Ivan

I’d propose to aim for working nightlies built on Wednesday 1 February.
We most likely want a day to fix any regressions that might have slipped
in so we need to merge something to Foreman develop on Monday 30
January. Since reviewers need time to review that means the ActiveJob PR
needs to be there some time in advance. I can’t speak for the reviewers,
but I’m guessing before Monday morning would be nice.

If no consensus on the ActiveJob PR can be reached by Monday evening,
then the reversal is merged instead. This still allows to verify/fix
other regressions on Tuesday and have working nightlies on Wednesday.

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[iNecas]

Ohad Levy <ohadlevy@gmail.com> writes:

>
>> Ewoud Kohl van Wijngaarden <ewoud@kohlvanwijngaarden.nl> writes:
>>
>> >> Ohad Levy <ohadlevy@gmail.com> writes:
>> >>
>> >> >
>> >> >> Why we haven't considered ActiveJob API in the first place? This
>> looks
>> >> >> like ideal solution, easy development and documented API.
>> >> >>
>> >> >> Can't ActiveJob help us with memory issues we have with external task
>> >> >> executor in Katello? I mean what if we run background tasks inside
>> >> >> passenger process by default for easier deployment and maintenance.
>> Of
>> >> >> course, restart would mean lost queue.
>> >> >>
>> >> >
>> >> > I don't know if that's what you had in mind or not, but if it end up
>> being
>> >> > blocking the request, its not useful at all, as that would lead to
>> more
>> >> > confusion, and a developer will need to care for more usage cases (is
>> it
>> >> > blocking now, is it running async) and especially when there is a
>> need for
>> >> > ui, imho it just makes it harder.
>> >> >
>> >> > if we do move to active job api in core(+1), I could see us starting
>> >> > without tasks first for pure async operations, over time converting
>> dynflow
>> >> > to be orchestration engine vs async and orchestration.
>> >> > Futher, I would challenge a bit the orchestration implementation in
>> Katello
>> >> > (not in a negative way) but if a task fails due to unavailability of
>> one of
>> >> > the backends, the application should know how to deal with it (micro
>> >> > service / soa) rather then fail / lock the task for the future, I
>> would ask
>> >> > if this is the correct long term approach for complex operations.
>> >>
>> >> While I like challenging the things, let's not make this thread to
>> >> broad.
>> >>
>> >> I've made quite a progress with ActiveJob, and while it's not perfect,
>> >> for basic introduction of async operations it should work well enough.
>> >>
>> >> However, I'm still few days from having it ready, and I want keep the
>> >> promise to propose the solution to falling nightlies by Monday evening.
>> >> Therefore, I've opened a PR for reverting the original change
>> >>
>> >> Refs #15779 - make background processing unavailable for now by iNecas · Pull Request #4217 · theforeman/foreman · GitHub
>> >>
>> >> This is not a resignation of trying to get the tasks functionality into
>> >> core, but rather lowering a pressure a bit so that we can focus on the
>> >> right things to do. I hope the PR adding the functionality back, using
>> >> the approach we talked about, will come sooner rather than later,
>> followed
>> >> by adding the tasking plumbing into core, to support async operations,
>> >> as well as dynflow orchestration and current host orchestration.
>> >
>> > I greatly appreciate your effort to fix the situation.
>> >
>> > Given nightly has already been broken for a bit, I'd find it acceptable
>> > if we pushed the reversal to next week. That should give you a little
>> > bit extra time to work on ActiveJob which is most likely where we want
>> > to end up anyway.
>> >
>> > My biggest question is: in a few days (I'm guessing that's 3 or 4), will
>> > it be in mergeable shape? Would that mean we can expect it to be merged
>> > on Monday? With FOSDEM and cfgmgtcamp coming up I'd like to have nightly
>> > back into shape before it.
>>
>> I think it's realistic. If it's acceptable for people to wait for few
>> days, I'm ok with the updated plan.
>>
>
> TBH I dont fully understand the solution.
>
> we will not ship foreman-tasks by default, but rather allow a developer to
> write in a foreman-task compatible way (e.g. can work with and without
> foreman tasks), and merge foreman tasks into core at some time in the
> future.
>
> What I don't get is:
> 1. if there is no foreman tasks, then regardless of how i execute (e.g. via
> active job) it would still be blocking, UI would need to account for that -
> meaning that we still going to have to modify the code that uses it.
> 2. we are going to remove foreman-tasks from core for the time being
> anyway, so why wait on the revert? tbh, report importing in the background
> is really low prio, and puppet class importing while useful, we can live
> without just as we did for 6 years or so before.
>
> so unless I miss understand something, I would vote to remove it now to get
> nightly back.

As I said, I don't really mind going one way or another. I think
if we revert or not doesn't have much influence on the result of
the next PR. If there are doubts, perhaps it's better to revert now (to
unblock stabilization), so that we are not pressed by the blocked build
thought the additional discussions.

– Ivan

···

> On Tue, Jan 24, 2017 at 9:51 AM, Ivan Necas wrote: >> > On Tue, Jan 24, 2017 at 12:10:34AM +0100, Ivan Necas wrote: >> >> > On Fri, Jan 20, 2017 at 7:14 PM, Lukas Zapletal >> wrote:

Ohad

– Ivan

I’d propose to aim for working nightlies built on Wednesday 1 February.
We most likely want a day to fix any regressions that might have slipped
in so we need to merge something to Foreman develop on Monday 30
January. Since reviewers need time to review that means the ActiveJob PR
needs to be there some time in advance. I can’t speak for the reviewers,
but I’m guessing before Monday morning would be nice.

If no consensus on the ActiveJob PR can be reached by Monday evening,
then the reversal is merged instead. This still allows to verify/fix
other regressions on Tuesday and have working nightlies on Wednesday.

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[lzap]

I like the idea of having plain ActiveJob for simple background tasks
and providing tasks/dynflow for complex orchestration processes (also
available via ActiveJob API). While I understand whole motivation why
dynflow/task was created in the first place (providing a way to do
one-way orchestrations that can be fixed by users), this could open
doors to rewriting some non-critical Katello operations via simple
ActiveJob/Ruby (if there are any - I think calculate errata is
repeatable action that goes only to Pulp to name a candidate).

I think this gives us the flexibility of both worlds - simple
background task that either succeeds or fails (and its deleted from
queue) or complex orchestration process written in dynflow for things
like register a system in katello (multiple backends involved).

Ivan, what do you think about providing non-foreman task interface via
ActiveJob API?

LZ

···

On Tue, Jan 24, 2017 at 5:51 PM, Ivan Necas wrote: > Ohad Levy writes: > >> On Tue, Jan 24, 2017 at 9:51 AM, Ivan Necas wrote: >> >>> Ewoud Kohl van Wijngaarden writes: >>> >>> > On Tue, Jan 24, 2017 at 12:10:34AM +0100, Ivan Necas wrote: >>> >> Ohad Levy writes: >>> >> >>> >> > On Fri, Jan 20, 2017 at 7:14 PM, Lukas Zapletal >>> wrote: >>> >> > >>> >> >> Why we haven't considered ActiveJob API in the first place? This >>> looks >>> >> >> like ideal solution, easy development and documented API. >>> >> >> >>> >> >> Can't ActiveJob help us with memory issues we have with external task >>> >> >> executor in Katello? I mean what if we run background tasks inside >>> >> >> passenger process by default for easier deployment and maintenance. >>> Of >>> >> >> course, restart would mean lost queue. >>> >> >> >>> >> > >>> >> > I don't know if that's what you had in mind or not, but if it end up >>> being >>> >> > blocking the request, its not useful at all, as that would lead to >>> more >>> >> > confusion, and a developer will need to care for more usage cases (is >>> it >>> >> > blocking now, is it running async) and especially when there is a >>> need for >>> >> > ui, imho it just makes it harder. >>> >> > >>> >> > if we do move to active job api in core(+1), I could see us starting >>> >> > without tasks first for pure async operations, over time converting >>> dynflow >>> >> > to be orchestration engine vs async and orchestration. >>> >> > Futher, I would challenge a bit the orchestration implementation in >>> Katello >>> >> > (not in a negative way) but if a task fails due to unavailability of >>> one of >>> >> > the backends, the application should know how to deal with it (micro >>> >> > service / soa) rather then fail / lock the task for the future, I >>> would ask >>> >> > if this is the correct long term approach for complex operations. >>> >> >>> >> While I like challenging the things, let's not make this thread to >>> >> broad. >>> >> >>> >> I've made quite a progress with ActiveJob, and while it's not perfect, >>> >> for basic introduction of async operations it should work well enough. >>> >> >>> >> However, I'm still few days from having it ready, and I want keep the >>> >> promise to propose the solution to falling nightlies by Monday evening. >>> >> Therefore, I've opened a PR for reverting the original change >>> >> >>> >> https://github.com/theforeman/foreman/pull/4217 >>> >> >>> >> This is not a resignation of trying to get the tasks functionality into >>> >> core, but rather lowering a pressure a bit so that we can focus on the >>> >> right things to do. I hope the PR adding the functionality back, using >>> >> the approach we talked about, will come sooner rather than later, >>> followed >>> >> by adding the tasking plumbing into core, to support async operations, >>> >> as well as dynflow orchestration and current host orchestration. >>> > >>> > I greatly appreciate your effort to fix the situation. >>> > >>> > Given nightly has already been broken for a bit, I'd find it acceptable >>> > if we pushed the reversal to next week. That should give you a little >>> > bit extra time to work on ActiveJob which is most likely where we want >>> > to end up anyway. >>> > >>> > My biggest question is: in a few days (I'm guessing that's 3 or 4), will >>> > it be in mergeable shape? Would that mean we can expect it to be merged >>> > on Monday? With FOSDEM and cfgmgtcamp coming up I'd like to have nightly >>> > back into shape before it. >>> >>> I think it's realistic. If it's acceptable for people to wait for few >>> days, I'm ok with the updated plan. >>> >> >> TBH I dont fully understand the solution. >> >> we will not ship foreman-tasks by default, but rather allow a developer to >> write in a foreman-task compatible way (e.g. can work with and without >> foreman tasks), and merge foreman tasks into core at some time in the >> future. >> >> What I don't get is: >> 1. if there is no foreman tasks, then regardless of how i execute (e.g. via >> active job) it would still be blocking, UI would need to account for that - >> meaning that we still going to have to modify the code that uses it. >> 2. we are going to remove foreman-tasks from core for the time being >> anyway, so why wait on the revert? tbh, report importing in the background >> is really low prio, and puppet class importing while useful, we can live >> without just as we did for 6 years or so before. >> >> so unless I miss understand something, I would vote to remove it now to get >> nightly back. > > As I said, I don't really mind going one way or another. I think > if we revert or not doesn't have much influence on the result of > the next PR. If there are doubts, perhaps it's better to revert now (to > unblock stabilization), so that we are not pressed by the blocked build > thought the additional discussions. > > -- Ivan > >> >> Ohad >> >>> >>> -- Ivan >>> >>> > >>> > I'd propose to aim for working nightlies built on Wednesday 1 February. >>> > We most likely want a day to fix any regressions that might have slipped >>> > in so we need to merge something to Foreman develop on Monday 30 >>> > January. Since reviewers need time to review that means the ActiveJob PR >>> > needs to be there some time in advance. I can't speak for the reviewers, >>> > but I'm guessing before Monday morning would be nice. >>> > >>> > If no consensus on the ActiveJob PR can be reached by Monday evening, >>> > then the reversal is merged instead. This still allows to verify/fix >>> > other regressions on Tuesday and have working nightlies on Wednesday. >>> > >>> > -- >>> > You received this message because you are subscribed to the Google >>> Groups "foreman-dev" group. >>> > To unsubscribe from this group and stop receiving emails from it, send >>> an email to foreman-dev+unsubscribe@googlegroups.com. >>> > For more options, visit https://groups.google.com/d/optout. >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "foreman-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to foreman-dev+unsubscribe@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas @lzap Zapletal

[iNecas]

> I like the idea of having plain ActiveJob for simple background tasks
> and providing tasks/dynflow for complex orchestration processes (also
> available via ActiveJob API). While I understand whole motivation why
> dynflow/task was created in the first place (providing a way to do
> one-way orchestrations that can be fixed by users), this could open
> doors to rewriting some non-critical Katello operations via simple
> ActiveJob/Ruby (if there are any - I think calculate errata is
> repeatable action that goes only to Pulp to name a candidate).
>
> I think this gives us the flexibility of both worlds - simple
> background task that either succeeds or fails (and its deleted from
> queue) or complex orchestration process written in dynflow for things
> like register a system in katello (multiple backends involved).
>
> Ivan, what do you think about providing non-foreman task interface via
> ActiveJob API?

That is the plan.

In the mean time, the original changes were removed: the nightlies should
start working again soon

– Ivan

···

On Wed, Jan 25, 2017 at 12:38 PM, Lukas Zapletal wrote:

LZ

On Tue, Jan 24, 2017 at 5:51 PM, Ivan Necas inecas@redhat.com wrote:

Ohad Levy ohadlevy@gmail.com writes:

On Tue, Jan 24, 2017 at 9:51 AM, Ivan Necas inecas@redhat.com wrote:

Ewoud Kohl van Wijngaarden ewoud@kohlvanwijngaarden.nl writes:

On Tue, Jan 24, 2017 at 12:10:34AM +0100, Ivan Necas wrote:

Ohad Levy ohadlevy@gmail.com writes:

On Fri, Jan 20, 2017 at 7:14 PM, Lukas Zapletal lzap@redhat.com >>>> wrote:

Why we haven’t considered ActiveJob API in the first place? This
looks
like ideal solution, easy development and documented API.

Can’t ActiveJob help us with memory issues we have with external task
executor in Katello? I mean what if we run background tasks inside
passenger process by default for easier deployment and maintenance.
Of
course, restart would mean lost queue.

I don’t know if that’s what you had in mind or not, but if it end up
being
blocking the request, its not useful at all, as that would lead to
more
confusion, and a developer will need to care for more usage cases (is
it
blocking now, is it running async) and especially when there is a
need for
ui, imho it just makes it harder.

if we do move to active job api in core(+1), I could see us starting
without tasks first for pure async operations, over time converting
dynflow
to be orchestration engine vs async and orchestration.
Futher, I would challenge a bit the orchestration implementation in
Katello
(not in a negative way) but if a task fails due to unavailability of
one of
the backends, the application should know how to deal with it (micro
service / soa) rather then fail / lock the task for the future, I
would ask
if this is the correct long term approach for complex operations.

While I like challenging the things, let’s not make this thread to
broad.

I’ve made quite a progress with ActiveJob, and while it’s not perfect,
for basic introduction of async operations it should work well enough.

However, I’m still few days from having it ready, and I want keep the
promise to propose the solution to falling nightlies by Monday evening.
Therefore, I’ve opened a PR for reverting the original change

Refs #15779 - make background processing unavailable for now by iNecas · Pull Request #4217 · theforeman/foreman · GitHub

This is not a resignation of trying to get the tasks functionality into
core, but rather lowering a pressure a bit so that we can focus on the
right things to do. I hope the PR adding the functionality back, using
the approach we talked about, will come sooner rather than later,
followed
by adding the tasking plumbing into core, to support async operations,
as well as dynflow orchestration and current host orchestration.

I greatly appreciate your effort to fix the situation.

Given nightly has already been broken for a bit, I’d find it acceptable
if we pushed the reversal to next week. That should give you a little
bit extra time to work on ActiveJob which is most likely where we want
to end up anyway.

My biggest question is: in a few days (I’m guessing that’s 3 or 4), will
it be in mergeable shape? Would that mean we can expect it to be merged
on Monday? With FOSDEM and cfgmgtcamp coming up I’d like to have nightly
back into shape before it.

I think it’s realistic. If it’s acceptable for people to wait for few
days, I’m ok with the updated plan.

TBH I dont fully understand the solution.

we will not ship foreman-tasks by default, but rather allow a developer to
write in a foreman-task compatible way (e.g. can work with and without
foreman tasks), and merge foreman tasks into core at some time in the
future.

What I don’t get is:

1. if there is no foreman tasks, then regardless of how i execute (e.g. via
   active job) it would still be blocking, UI would need to account for that -
   meaning that we still going to have to modify the code that uses it.
2. we are going to remove foreman-tasks from core for the time being
   anyway, so why wait on the revert? tbh, report importing in the background
   is really low prio, and puppet class importing while useful, we can live
   without just as we did for 6 years or so before.

so unless I miss understand something, I would vote to remove it now to get
nightly back.

As I said, I don’t really mind going one way or another. I think
if we revert or not doesn’t have much influence on the result of
the next PR. If there are doubts, perhaps it’s better to revert now (to
unblock stabilization), so that we are not pressed by the blocked build
thought the additional discussions.

– Ivan

Ohad

– Ivan

I’d propose to aim for working nightlies built on Wednesday 1 February.
We most likely want a day to fix any regressions that might have slipped
in so we need to merge something to Foreman develop on Monday 30
January. Since reviewers need time to review that means the ActiveJob PR
needs to be there some time in advance. I can’t speak for the reviewers,
but I’m guessing before Monday morning would be nice.

If no consensus on the ActiveJob PR can be reached by Monday evening,
then the reversal is merged instead. This still allows to verify/fix
other regressions on Tuesday and have working nightlies on Wednesday.

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas @lzap Zapletal

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.