Moments of Coffee Meeting: Fog and Compute Resrouces NG

Development
#1

Hey,

I brought an interesting topic about RHEV support in Foreman, Ori
pointed out that RHEV v3 API is going to be deprecated soon (rumors
are 4.2 which is very soon) and since we are still using v3 via
rbovirt, it's the time to start dicussion in moving towards v4.

We switched to discussing the idea to implement RHEV v4 support via
newly created API that would allow us easily creating new CR
providers, Fog based or non-Fog based. Concerns were obvious - we do
not want to rewrite Fog or big part of it, on the other hand Marek
pointed out we want very simple API which will be pure Ruby (and not
XML hell) as minimal as it can be.

Marek pointed out that our API could include things like memory sizes,
pool sizes and other lists (OS lists) which are spread around in Fog,
but we want consistent UX. Our API could be a chance to unify these.
Also our approach can increase performance as we not always need all
flags and data.

Shim brought an idea to use Facets for the new CR (CRNG - new
generation) which would allow to run multiple versions of compute
resources easily which is something we will need to do for RHEV (we
still want to support oVirt 3.x for some time).

He also raised concern in going too far in unification, Marek answered
that we should only unify internal things and primarily data (memory).
Greg added that UI is added value and not core goal. He also mentioned
that Fog project lost its momentum a bit and it is not very relevant
to us these days.

I brought an idea to solve our long-standing request to proxy CR
communication through proxy, we should keep that in mind when
designing the new API. Which should be definitely as minimal as
possible to allow writing non-Fog providers. Also one problem we could
tackle is to stop doing long HTTP requests from web request thead, we
could consider using Rails Jobs or Foreman Tasks for tasks.

We agreed that we are not getting rid of Fog anytime soon, but we
would love to have the possibility not to use Fog in the future. This
is a good candidate for RFC and we are running out of time thank to
RHEV v3 API deprecation.

··· -- Later, Lukas @lzap Zapletal
#2

> Hey,
>
> I brought an interesting topic about RHEV support in Foreman, Ori
> pointed out that RHEV v3 API is going to be deprecated soon (rumors
> are 4.2 which is very soon) and since we are still using v3 via
> rbovirt, it's the time to start dicussion in moving towards v4.
>
> We switched to discussing the idea to implement RHEV v4 support via
> newly created API that would allow us easily creating new CR
> providers, Fog based or non-Fog based. Concerns were obvious - we do
> not want to rewrite Fog or big part of it, on the other hand Marek
> pointed out we want very simple API which will be pure Ruby (and not
> XML hell) as minimal as it can be.
>
Which XML hell do you currently have? granted v3 of ovirt api uses XML, but
that's surely not related to fog.

> Marek pointed out that our API could include things like memory sizes,
> pool sizes and other lists (OS lists) which are spread around in Fog,
> but we want consistent UX. Our API could be a chance to unify these.
> Also our approach can increase performance as we not always need all
> flags and data.
>
> Shim brought an idea to use Facets for the new CR (CRNG - new
> generation) which would allow to run multiple versions of compute
> resources easily which is something we will need to do for RHEV (we
> still want to support oVirt 3.x for some time).
>
that would be nice, especially if we can separate workers (and not load all
into every process memory).

>
> He also raised concern in going too far in unification, Marek answered
> that we should only unify internal things and primarily data (memory).
> Greg added that UI is added value and not core goal. He also mentioned
> that Fog project lost its momentum a bit and it is not very relevant
> to us these days.
>
> I would disagree around UI, IMHO we need to strive and improve are UI,
imho most people who leave foreman do so because of a poor UX, or lack of
relevance in new set of techonologies ( cloud born companies, containers
only etc).

> I brought an idea to solve our long-standing request to proxy CR
> communication through proxy, we should keep that in mind when
> designing the new API. Which should be definitely as minimal as
> possible to allow writing non-Fog providers. Also one problem we could
> tackle is to stop doing long HTTP requests from web request thead, we
> could consider using Rails Jobs or Foreman Tasks for tasks.
>

TBH I do believe that fog actually supports proxy (excon and rest client do
which imho are the majority of http connections).

>
> We agreed that we are not getting rid of Fog anytime soon, but we
> would love to have the possibility not to use Fog in the future. This
> is a good candidate for RFC and we are running out of time thank to
> RHEV v3 API deprecation.
>

Since we get a lot of lift from fog, especially for popular providers (e.g.
ec2) IMHO its not a good idea to remove fog, which means that we balance
between community contributions to fog (e.g. stuff we won't "enjoy" as we
will not corporate) vs other benefits mentioned above.

I for one, is not convinced the effort to not run with fog is less work
then adding / updating a fog provider.

Ohad

··· On Mon, May 22, 2017 at 12:30 PM, Lukas Zapletal wrote:

–
Later,
Lukas @lzap Zapletal

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#3

I do agree with Ohad here. I'd focus on improving the fog-providers and not trying to reinvent the weel.
I think, "cloud" topics like focussing on real server orchestration (think hostgroup as an auto scaling group) adds more benefits for a user. A user usually doesn't care about the library used. Using foreman as a tool to setup a cloud or container environment on bare metal does add value.

Fog doesn't do any network orchestration, yet. If we add this (e.g. provision a vlan on a switchport or some SDN), that would be a valid point imho to switch the library.

  • Timo
··· > On 22. May 2017, at 12:27, Ohad Levy wrote: > > Since we get a lot of lift from fog, especially for popular providers (e.g. ec2) IMHO its not a good idea to remove fog, which means that we balance between community contributions to fog (e.g. stuff we won't "enjoy" as we will not corporate) vs other benefits mentioned above. > > I for one, is not convinced the effort to not run with fog is less work then adding / updating a fog provider. > > Ohad
#4

Timo Goebel <mail@timogoebel.name> writes:

>>
>> Since we get a lot of lift from fog, especially for popular providers (e.g. ec2) IMHO its not a good idea to remove fog, which means that we balance between community contributions to fog (e.g. stuff we won't "enjoy" as we will not corporate) vs other benefits mentioned above.
>>
>> I for one, is not convinced the effort to not run with fog is less work then adding / updating a fog provider.
>>
>> Ohad
>
> I do agree with Ohad here. I'd focus on improving the fog-providers and not trying to reinvent the weel.
> I think, "cloud" topics like focussing on real server orchestration (think hostgroup as an auto scaling group) adds more benefits for a user. A user usually doesn't care about the library used. Using foreman as a tool to setup a cloud or container environment on bare metal does add value.
>
> Fog doesn't do any network orchestration, yet. If we add this (e.g.
> provision a vlan on a switchport or some SDN), that would be a valid
> point imho to switch the library.

I think what's missing is not well-defined layer between Foreman and
Fog, and we perhaps over-use the Fog as the unification layer.

I feel quite a lot of risk in connecting the migration between oVirt v3
vs. v4 with introduction of new layer into the Foreman to serve better
for our purposes, that would not use the Fog. Especially when the APIv4
is pretty close.

If we make fog oVirt provider use APIv4, we still have chance that the
code will be used outside of Foreman, that will make the implementation
better: from the past - did we experience this happening in oVirt or
other providers? (my feeling is that they did, but not monitoring that
that much)

Long story short:

  1. I agree we should not expose Fog as THE layer we use in the UI/API,
    to add the entry points that are interesting for use but not for fog
    that much

  2. we should continue with fog-ovirt to use APIv4

Even though it looks compelling to connecting this two, I would rather
take it independent steps.

– Ivan

··· >> On 22. May 2017, at 12:27, Ohad Levy wrote:
  • Timo

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#5

> Timo Goebel <mail@timogoebel.name> writes:
>
>>>
>>> Since we get a lot of lift from fog, especially for popular providers (e.g. ec2) IMHO its not a good idea to remove fog, which means that we balance between community contributions to fog (e.g. stuff we won't "enjoy" as we will not corporate) vs other benefits mentioned above.
>>>
>>> I for one, is not convinced the effort to not run with fog is less work then adding / updating a fog provider.
>>>
>>> Ohad
>>
>> I do agree with Ohad here. I'd focus on improving the fog-providers and not trying to reinvent the weel.
>> I think, "cloud" topics like focussing on real server orchestration (think hostgroup as an auto scaling group) adds more benefits for a user. A user usually doesn't care about the library used. Using foreman as a tool to setup a cloud or container environment on bare metal does add value.
>>
>> Fog doesn't do any network orchestration, yet. If we add this (e.g.
>> provision a vlan on a switchport or some SDN), that would be a valid
>> point imho to switch the library.
>
>
> I think what's missing is not well-defined layer between Foreman and
> Fog, and we perhaps over-use the Fog as the unification layer.
>
> I feel quite a lot of risk in connecting the migration between oVirt v3
> vs. v4 with introduction of new layer into the Foreman to serve better
> for our purposes, that would not use the Fog. Especially when the APIv4
> is pretty close.
>
> If we make fog oVirt provider use APIv4, we still have chance that the
> code will be used outside of Foreman, that will make the implementation
> better: from the past - did we experience this happening in oVirt or
> other providers? (my feeling is that they did, but not monitoring that
> that much)
>
> Long story short:
>
> 1. I agree we should not expose Fog as THE layer we use in the UI/API,
> to add the entry points that are interesting for use but not for fog
> that much

+1 for adding a layer between our UI/API and fog or potentially some
other library we might decide to use (or write) in future. I think
it's an important step, regardless of whether we decide to use fog for
oVirt v4 api or not.

··· On Tue, May 23, 2017 at 10:07 AM, Ivan Necas wrote: >>> On 22. May 2017, at 12:27, Ohad Levy wrote:
  1. we should continue with fog-ovirt to use APIv4

Even though it looks compelling to connecting this two, I would rather
take it independent steps.

– Ivan

  • Timo

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#6

I was typing quickly, maybe it was confusing, but nobody said we aim
to replace fog at all. We were discussing decoupling from Fog API
inside Foreman so we could enable writing non-Fog providers, which led
to more opened topics like facets, smart-proxy communication or
dynflow. This would be always complementary to fog.

Ivan pointed out well that RHEV v4 might not be good candidate for
such an implementation because we will likely be backporting this,
valid one.

I just made these notes in case someone wants to take chance and start
a patch or writing a RFE.

··· On Tue, May 23, 2017 at 10:28 AM, Tomas Strachota wrote: > On Tue, May 23, 2017 at 10:07 AM, Ivan Necas wrote: >> Timo Goebel writes: >> >>>> On 22. May 2017, at 12:27, Ohad Levy wrote: >>>> >>>> Since we get a lot of lift from fog, especially for popular providers (e.g. ec2) IMHO its not a good idea to remove fog, which means that we balance between community contributions to fog (e.g. stuff we won't "enjoy" as we will not corporate) vs other benefits mentioned above. >>>> >>>> I for one, is not convinced the effort to not run with fog is less work then adding / updating a fog provider. >>>> >>>> Ohad >>> >>> I do agree with Ohad here. I'd focus on improving the fog-providers and not trying to reinvent the weel. >>> I think, "cloud" topics like focussing on real server orchestration (think hostgroup as an auto scaling group) adds more benefits for a user. A user usually doesn't care about the library used. Using foreman as a tool to setup a cloud or container environment on bare metal does add value. >>> >>> Fog doesn't do any network orchestration, yet. If we add this (e.g. >>> provision a vlan on a switchport or some SDN), that would be a valid >>> point imho to switch the library. >> >> >> I think what's missing is not well-defined layer between Foreman and >> Fog, and we perhaps over-use the Fog as the unification layer. >> >> I feel quite a lot of risk in connecting the migration between oVirt v3 >> vs. v4 with introduction of new layer into the Foreman to serve better >> for our purposes, that would not use the Fog. Especially when the APIv4 >> is pretty close. >> >> If we make fog oVirt provider use APIv4, we still have chance that the >> code will be used outside of Foreman, that will make the implementation >> better: from the past - did we experience this happening in oVirt or >> other providers? (my feeling is that they did, but not monitoring that >> that much) >> >> Long story short: >> >> 1. I agree we should not expose Fog as THE layer we use in the UI/API, >> to add the entry points that are interesting for use but not for fog >> that much > > +1 for adding a layer between our UI/API and fog or potentially some > other library we might decide to use (or write) in future. I think > it's an important step, regardless of whether we decide to use fog for > oVirt v4 api or not. > >> >> 2. we should continue with fog-ovirt to use APIv4 >> >> Even though it looks compelling to connecting this two, I would rather >> take it independent steps. >> >> -- Ivan >> >>> >>> - Timo >>> >>> -- >>> You received this message because you are subscribed to the Google Groups "foreman-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.

–
Later,
Lukas @lzap Zapletal