FYI Katello CI is broken

Strongly disagree to suggestion that developers integrate against anything but latest develop/master in github. Having foreman work tested against a stale katello, or katello work against stale foreman, is just asking for massive pain.

Core developers (which I mean to include non-community folks working on both foreman, katello, and any of the product plugins) need to understand how they fit into the product ecosystem. We are no longer just katello or just foreman devs and should take the necessary responsibility of knowing that a change in one half of the core can impact the other. The safety measure of all product tests being run against all product pull-requests is absolutely needed, in my opinion.

I understand there is some pain to now having to work in a much larger sandbox, but this is not optional. If a pull-request on one half impacts the second half, then there should be two pull-requests that together make things work.

Why is this even being discussed? Does anyone from the core product disagree? Will this hurt the community developers in any way?

··· ----- Original Message ----- > From: "Bryan Kearney" > To: foreman-dev@googlegroups.com > Sent: Thursday, February 20, 2014 7:48:16 AM > Subject: Re: [foreman-dev] FYI Katello CI is broken > > On 02/20/2014 03:53 AM, Petr Chalupa wrote: > > > > > > On 19.02.14 20:26, Ivan Necas wrote: > >> Even short-term it solved the issue just partly: the nightlies get > >> broken as well. > >> > >> We've been there few months ago (and similarly with pulp and candlepin > >> long time before) and > >> we alwys ended up with Katello decinding when is time to update the deps > >> (and the issues were resolved on the time when the update happened). > >> > >> We could probably have a branch in Foreman > >> repo that would point to the sha that we are targetting against. > >> But it would probably also mean maintaing the repo holding tha > >> foreman packages known to work with Katello. The downside is > >> the risk of breaking changes landing into Foreman and Katello > >> learning too late about that. Also, the burden of mainaing > >> different set of repos could be painful. > >> > >> I would probably like more with CI running katello (and other plugins) > >> tests with every PR in Foreman, indicating, if it's breaking or not > >> and agreeing on the proceedure when some incombatible change is > >> introduced: > >> if it's going to be merged anyway, would be nice to have the patch for > >> Katello > >> prepared, so that we can just update simultaniously. > > > > +1 run Katello tests on each Foreman PR > My suggestion would be that daily workflows be against a nightly or 2 > day old build. That way one nightlu job looks for "core breaks plugin" > and a second CI supports the PR process. When a build passes the > nightly, it can get moved to the PR build. That way, we catch core > breakage but those do not stop active development > > -- bk > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. >

Massive depends on how stale. 1-2 days… I doubt the pain is massive.
Building against 1.4 or 1.3, I agree.

– bk

··· On 02/20/2014 08:51 AM, Tom McKay wrote: > > > ----- Original Message ----- >> From: "Bryan Kearney" >> To: foreman-dev@googlegroups.com >> Sent: Thursday, February 20, 2014 7:48:16 AM >> Subject: Re: [foreman-dev] FYI Katello CI is broken >> >> On 02/20/2014 03:53 AM, Petr Chalupa wrote: >>> >>> >>> On 19.02.14 20:26, Ivan Necas wrote: >>>> Even short-term it solved the issue just partly: the nightlies get >>>> broken as well. >>>> >>>> We've been there few months ago (and similarly with pulp and candlepin >>>> long time before) and >>>> we alwys ended up with Katello decinding when is time to update the deps >>>> (and the issues were resolved on the time when the update happened). >>>> >>>> We could probably have a branch in Foreman >>>> repo that would point to the sha that we are targetting against. >>>> But it would probably also mean maintaing the repo holding tha >>>> foreman packages known to work with Katello. The downside is >>>> the risk of breaking changes landing into Foreman and Katello >>>> learning too late about that. Also, the burden of mainaing >>>> different set of repos could be painful. >>>> >>>> I would probably like more with CI running katello (and other plugins) >>>> tests with every PR in Foreman, indicating, if it's breaking or not >>>> and agreeing on the proceedure when some incombatible change is >>>> introduced: >>>> if it's going to be merged anyway, would be nice to have the patch for >>>> Katello >>>> prepared, so that we can just update simultaniously. >>> >>> +1 run Katello tests on each Foreman PR >> My suggestion would be that daily workflows be against a nightly or 2 >> day old build. That way one nightlu job looks for "core breaks plugin" >> and a second CI supports the PR process. When a build passes the >> nightly, it can get moved to the PR build. That way, we catch core >> breakage but those do not stop active development >> >> -- bk >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > Strongly disagree to suggestion that developers integrate against anything but latest develop/master in github. Having foreman work tested against a stale katello, or katello work against stale foreman, is just asking for massive pain.

Potentially yes. Right now, if a PR to Foreman Core breaks Core's
tests, the author (community or not) is expected to deal with it. Are
you saying that every contributor needs to run the full katello stack
and take ownership of issues in both codebases (even on distros where
Katello will not run)?. Are we sayng that the Katello Devs will watch
the test jobs and provide feedback/patches to the contributor? Are we
saying it's the person doing the merge that needs to be aware of
possible issues in the plugins? I don't think anyone disagrees that
these tests need to happen, it's the changes to the process that need
to be clearly defined.

I'd suggest that both the second and third option are little better
than what we have right now (as it's a human-check on the status, and
thus subject to human error), and the first option will indeed impact
community contributors. What Bryan is suggesting seems a sensible
compromise to me. I think we might even be able to automate the update
of the known-good SHAsum with a little Jenkins scripting.

··· On 20 February 2014 13:51, Tom McKay wrote: > Why is this even being discussed? Does anyone from the core product disagree? Will this hurt the community developers in any way?

By not running master against master on every pull-request, doesn't that just delay the point where you figure out things are broken? And doesn't that also mean that several merged pull-requests could pile up between the offending merge and when it is discovered?

Yes, I am saying every contributor with an email ending in @redhat.com needs to "run the full katello stack and take ownership of issues in both codebases." Taking ownership means coordinating with the necessary resources to solve the issue.

For community members that make pull-requests, I'd expect the @redhat.com pull-request helpers to take responsibility. Isn't that how it works now in foreman community? There is a foreman veteran that tests the p-r, offers suggestions, and even cleans up the commit prior to merge. Do I expect the foreman community to know about katello? No. Should I expect the p-r reviewer and merger on foretello team to know? Absolutely.

Again, I'm suggesting that absolutely no p-r gets merged into either core (foremand and katello) or a required plugin (discovery, etc.), that breaks the whole. To me that's the time to address the issues, prior to merge, not fire fighting after the fact.

··· ----- Original Message ----- > From: "Greg Sutcliffe" > To: "Foreman ." > Sent: Thursday, February 20, 2014 9:36:02 AM > Subject: Re: [foreman-dev] FYI Katello CI is broken > > On 20 February 2014 13:51, Tom McKay wrote: > > Why is this even being discussed? Does anyone from the core product > > disagree? Will this hurt the community developers in any way? > > Potentially yes. Right now, if a PR to Foreman Core breaks Core's > tests, the author (community or not) is expected to deal with it. Are > you saying that every contributor needs to run the full katello stack > and take ownership of issues in both codebases (even on distros where > Katello will not run)?. Are we sayng that the Katello Devs will watch > the test jobs and provide feedback/patches to the contributor? Are we > saying it's the person doing the merge that needs to be aware of > possible issues in the plugins? I don't think anyone disagrees that > these tests need to happen, it's the changes to the process that need > to be clearly defined. > > I'd suggest that both the second and third option are little better > than what we have right now (as it's a human-check on the status, and > thus subject to human error), and the first option will indeed impact > community contributors. What Bryan is suggesting seems a sensible > compromise to me. I think we might even be able to automate the update > of the known-good SHAsum with a little Jenkins scripting. > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/groups/opt_out. >

(speaking from my opinion)

>> Why is this even being discussed? Does anyone from the core product disagree? Will this hurt the community developers in any way?
> Potentially yes. Right now, if a PR to Foreman Core breaks Core's
> tests, the author (community or not) is expected to deal with it. Are
> you saying that every contributor needs to run the full katello stack
> and take ownership of issues in both codebases (even on distros where
> Katello will not run)?.
Not at all.

> Are we sayng that the Katello Devs will watch
> the test jobs and provide feedback/patches to the contributor?
Yes! Maybe not watch actively, but available to offer help and testing
against PRs where the katello tests fail. Foreman PRs are generally
picked through pretty well and not merged quickly. So i don't think
this would be a problem.

> Are we
> saying it's the person doing the merge that needs to be aware of
> possible issues in the plugins? I don't think anyone disagrees that
> these tests need to happen, it's the changes to the process that need
> to be clearly defined.
>
> I'd suggest that both the second and third option are little better
> than what we have right now (as it's a human-check on the status, and
> thus subject to human error), and the first option will indeed impact
> community contributors. What Bryan is suggesting seems a sensible
> compromise to me. I think we might even be able to automate the update
> of the known-good SHAsum with a little Jenkins scripting.
>
To me that's just delaying the pain and pushing the problem down the
road (even if its only two days). Fixing the problem earlier is much
much cheaper.

If the goal is to ensure that katello works with foreman and that
foreman works with katello having the katello tests run along side
foreman PRs is the only way to go IMHO. Especially for community
patches I could see the following happen:

  1. Community member submits PR
  2. User addresses comments and tests (to varying degrees)
  3. Tests are green
  4. Foreman committer cleans up patch and commits it.
  5. Katello breaks

who now has ownership of this breakage? The community contributor
probably won't have ownership as they may not use katello or be aware of
the breakage. A foreman developer may not feel that they have ownership
as they did not contribute the patch originally. So it would be up to a
katello developer to asses the breakage, determine the appropriate fix,
open a PR, and have it merged. Depending on the size of the issue it
could be a few days to a week (or more) to have it accepted into foreman
core. Meanwhile katello's nightly is stuck on some older version and
the devs have to know not to git pull (Or to stick at the magic working
sha).

If the katello's tests are running as part of this PR, I would see it
going very differently:

  1. Community member submits PR
  2. Katello tests break
  3. either:
    Katello team member works with user to get PR green or
    Katello team member works with foreman committer to clean up patch
    and make it green.
  4. Foreman committer commits it
  5. Katello still works and everybody is happy :slight_smile:

Would this cause PRs to take a little longer? Sure. How much longer?
Assuming katello devs are responsive and responsible it shouldn't take
that much longer. Will it save lots of frustration in the end? Yes :slight_smile:

-Justin

··· On 02/20/2014 09:36 AM, Greg Sutcliffe wrote: > On 20 February 2014 13:51, Tom McKay wrote:

> (speaking from my opinion)
>
>
>
>>
>>> Why is this even being discussed? Does anyone from the core product
>>> disagree? Will this hurt the community developers in any way?
>>>
>> Potentially yes. Right now, if a PR to Foreman Core breaks Core's
>> tests, the author (community or not) is expected to deal with it. Are
>> you saying that every contributor needs to run the full katello stack
>> and take ownership of issues in both codebases (even on distros where
>> Katello will not run)?.
>>
> Not at all.
>
>
> Are we sayng that the Katello Devs will watch
>> the test jobs and provide feedback/patches to the contributor?
>>
> Yes! Maybe not watch actively, but available to offer help and testing
> against PRs where the katello tests fail. Foreman PRs are generally picked
> through pretty well and not merged quickly. So i don't think this would be
> a problem.
>
>
> Are we
>> saying it's the person doing the merge that needs to be aware of
>> possible issues in the plugins? I don't think anyone disagrees that
>> these tests need to happen, it's the changes to the process that need
>> to be clearly defined.
>>
>> I'd suggest that both the second and third option are little better
>> than what we have right now (as it's a human-check on the status, and
>> thus subject to human error), and the first option will indeed impact
>> community contributors. What Bryan is suggesting seems a sensible
>> compromise to me. I think we might even be able to automate the update
>> of the known-good SHAsum with a little Jenkins scripting.
>>
>> To me that's just delaying the pain and pushing the problem down the
> road (even if its only two days). Fixing the problem earlier is much much
> cheaper.
>
> If the goal is to ensure that katello works with foreman and that foreman
> works with katello having the katello tests run along side foreman PRs is
> the only way to go IMHO. Especially for community patches I could see the
> following happen:
>
> 1. Community member submits PR
> 2. User addresses comments and tests (to varying degrees)
> 3. Tests are green
> 4. Foreman committer cleans up patch and commits it.
> 5. Katello breaks
>
> who now has ownership of this breakage? The community contributor
> probably won't have ownership as they may not use katello or be aware of
> the breakage. A foreman developer may not feel that they have ownership as
> they did not contribute the patch originally. So it would be up to a
> katello developer to asses the breakage, determine the appropriate fix,
> open a PR, and have it merged. Depending on the size of the issue it could
> be a few days to a week (or more) to have it accepted into foreman core.
> Meanwhile katello's nightly is stuck on some older version and the devs
> have to know not to git pull (Or to stick at the magic working sha).
>
> If the katello's tests are running as part of this PR, I would see it
> going very differently:
>
> 1. Community member submits PR
> 2. Katello tests break
> 3. either:
> Katello team member works with user to get PR green or
> Katello team member works with foreman committer to clean up patch and
> make it green.
>

^ I feel this part would need some automation and tooling around it to keep
track of and provide an easy view of all PRs that are currently breaking a
particular plugin in general to make referencing and viewing what needs
attention easier and more scalable.

Eric

··· On Thu, Feb 20, 2014 at 11:35 AM, Justin Sherrill wrote: > On 02/20/2014 09:36 AM, Greg Sutcliffe wrote: >> On 20 February 2014 13:51, Tom McKay wrote:
  1. Foreman committer commits it
  2. Katello still works and everybody is happy :slight_smile:

Would this cause PRs to take a little longer? Sure. How much longer?
Assuming katello devs are responsive and responsible it shouldn’t take
that much longer. Will it save lots of frustration in the end? Yes :slight_smile:

-Justin


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

> From: "Justin Sherrill" <jsherril@redhat.com>
> To: foreman-dev@googlegroups.com
> Sent: Thursday, February 20, 2014 11:35:07 AM
> Subject: Re: [foreman-dev] FYI Katello CI is broken
>
> (speaking from my opinion)
>
> >> Why is this even being discussed? Does anyone from the core product
> >> disagree? Will this hurt the community developers in any way?
> > Potentially yes. Right now, if a PR to Foreman Core breaks Core's
> > tests, the author (community or not) is expected to deal with it. Are
> > you saying that every contributor needs to run the full katello stack
> > and take ownership of issues in both codebases (even on distros where
> > Katello will not run)?.
> Not at all.
>
> > Are we sayng that the Katello Devs will watch
> > the test jobs and provide feedback/patches to the contributor?
> Yes! Maybe not watch actively, but available to offer help and testing
> against PRs where the katello tests fail. Foreman PRs are generally
> picked through pretty well and not merged quickly. So i don't think
> this would be a problem.
>
> > Are we
> > saying it's the person doing the merge that needs to be aware of
> > possible issues in the plugins? I don't think anyone disagrees that
> > these tests need to happen, it's the changes to the process that need
> > to be clearly defined.
> >
> > I'd suggest that both the second and third option are little better
> > than what we have right now (as it's a human-check on the status, and
> > thus subject to human error), and the first option will indeed impact
> > community contributors. What Bryan is suggesting seems a sensible
> > compromise to me. I think we might even be able to automate the update
> > of the known-good SHAsum with a little Jenkins scripting.
> >
> To me that's just delaying the pain and pushing the problem down the
> road (even if its only two days). Fixing the problem earlier is much
> much cheaper.
>
> If the goal is to ensure that katello works with foreman and that
> foreman works with katello having the katello tests run along side
> foreman PRs is the only way to go IMHO. Especially for community
> patches I could see the following happen:
>
> 1. Community member submits PR
> 2. User addresses comments and tests (to varying degrees)
> 3. Tests are green
> 4. Foreman committer cleans up patch and commits it.
> 5. Katello breaks
>
> who now has ownership of this breakage? The community contributor
> probably won't have ownership as they may not use katello or be aware of
> the breakage. A foreman developer may not feel that they have ownership
> as they did not contribute the patch originally. So it would be up to a
> katello developer to asses the breakage, determine the appropriate fix,
> open a PR, and have it merged. Depending on the size of the issue it
> could be a few days to a week (or more) to have it accepted into foreman
> core. Meanwhile katello's nightly is stuck on some older version and
> the devs have to know not to git pull (Or to stick at the magic working
> sha).
>
> If the katello's tests are running as part of this PR, I would see it
> going very differently:
>
> 1. Community member submits PR
> 2. Katello tests break
> 3. either:
> Katello team member works with user to get PR green or
> Katello team member works with foreman committer to clean up patch
> and make it green.
> 4. Foreman committer commits it
> 5. Katello still works and everybody is happy :slight_smile:
>
> Would this cause PRs to take a little longer? Sure. How much longer?
> Assuming katello devs are responsive and responsible it shouldn't take
> that much longer. Will it save lots of frustration in the end? Yes :slight_smile:
>
>
> -Justin
>

Another thing to note here is that the integration of Katello with Foreman is much tighter.
As we spend more time integrating Katello plugin with foreman base, its more likely that we'll be sending patches,
and expect foreman master to have them, so as to play well with Katello (and providing additional stability/functionality to foreman).
We need the 'master' in Katello & 'develop' in foreman to be in sync for this very reason.
I do not see adding a jenkins job - the same one that runs against Katello PRs - against foreman
being that much of a pain point. Delay it might but atleast we know it'll be more stable and not stall nightly builds for days.

Partha

··· ----- Original Message ----- > On 02/20/2014 09:36 AM, Greg Sutcliffe wrote: > > On 20 February 2014 13:51, Tom McKay wrote:


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I agree, we should test everything for each PR to have a chance (as
katello team) to fix the problems before not after CI and devel setup is
broken.

··· On 20.02.14 16:25, Tom McKay wrote: > > > ----- Original Message ----- >> From: "Greg Sutcliffe" >> To: "Foreman ." >> Sent: Thursday, February 20, 2014 9:36:02 AM >> Subject: Re: [foreman-dev] FYI Katello CI is broken >> >> On 20 February 2014 13:51, Tom McKay wrote: >>> Why is this even being discussed? Does anyone from the core product >>> disagree? Will this hurt the community developers in any way? >> >> Potentially yes. Right now, if a PR to Foreman Core breaks Core's >> tests, the author (community or not) is expected to deal with it. Are >> you saying that every contributor needs to run the full katello stack >> and take ownership of issues in both codebases (even on distros where >> Katello will not run)?. Are we sayng that the Katello Devs will watch >> the test jobs and provide feedback/patches to the contributor? Are we >> saying it's the person doing the merge that needs to be aware of >> possible issues in the plugins? I don't think anyone disagrees that >> these tests need to happen, it's the changes to the process that need >> to be clearly defined. >> >> I'd suggest that both the second and third option are little better >> than what we have right now (as it's a human-check on the status, and >> thus subject to human error), and the first option will indeed impact >> community contributors. What Bryan is suggesting seems a sensible >> compromise to me. I think we might even be able to automate the update >> of the known-good SHAsum with a little Jenkins scripting. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/groups/opt_out. >> > > By not running master against master on every pull-request, doesn't that just delay the point where you figure out things are broken? And doesn't that also mean that several merged pull-requests could pile up between the offending merge and when it is discovered? > > Yes, I am saying every contributor with an email ending in @redhat.com needs to "run the full katello stack and take ownership of issues in both codebases." Taking ownership means coordinating with the necessary resources to solve the issue. > > For community members that make pull-requests, I'd expect the @redhat.com pull-request helpers to take responsibility. Isn't that how it works now in foreman community? There is a foreman veteran that tests the p-r, offers suggestions, and even cleans up the commit prior to merge. Do I expect the foreman community to know about katello? No. Should I expect the p-r reviewer and merger on foretello team to know? Absolutely. > > Again, I'm suggesting that absolutely no p-r gets merged into either core (foremand and katello) or a required plugin (discovery, etc.), that breaks the whole. To me that's the time to address the issues, prior to merge, not fire fighting after the fact. >