Upcoming release schedules

Here's a summary of the upcoming activity in the three active release
streams (old/current/next).

I've also been rewriting the release management notes on the wiki,
starting with
http://projects.theforeman.org/projects/foreman/wiki/Release_Management
which now contains a rough guide to the policies I use for each release
among other things.

Foreman 1.8

··· =========== http://projects.theforeman.org/projects/foreman/wiki/Foreman_18_Schedule

I should have released 1.8.3 last month, but thought we’d have a few
more tickets. Currently there are only six, but I’ll release those as
1.8.3 in the next week. This will probably be the last 1.8 release if
there are no more important tickets closed in the next month or so.

Foreman 1.9

http://projects.theforeman.org/projects/foreman/wiki/Foreman_19_Schedule

RC3 was released on Friday and we’ve had confirmation that bug fixes for
earlier RC issues have been successful. No further bug reports, so
1.9.0 will probably be released on Monday.

Foreman 1.10

http://projects.theforeman.org/projects/foreman/wiki/Foreman_110_Schedule

Based on a mid August release of 1.9.0, I set the anticipated 1.10.0
date to the start of December. Branching will be mid-October, so we are
1.5 months into the 3.5 month development period.


Dominic Cleal
Red Hat Engineering

> Foreman 1.10
> ============
> Foreman 110 Schedule - Foreman
>
> Based on a mid August release of 1.9.0, I set the anticipated 1.10.0
> date to the start of December. Branching will be mid-October, so we are
> 1.5 months into the 3.5 month development period.

I was wondering what folks think about this pace. Is this too fast, too
slow? Are folks skipping version upgrades to lower amount of production
upgrades?

··· -- Later, Lukas #lzap Zapletal

This is delayed from today due to softwarecollections.org being
unavailable, so our package tests aren't able to run. Once there's news
on that, I'll plan it for another day this week.

https://www.redhat.com/archives/sclorg/2015-August/msg00009.html

The 1.9-stable branch is updated with two last fixes and translations,
so is waiting to be formally tagged.

··· On 12/08/15 12:14, Dominic Cleal wrote: > Foreman 1.9 > =========== > http://projects.theforeman.org/projects/foreman/wiki/Foreman_19_Schedule > > RC3 was released on Friday and we've had confirmation that bug fixes for > earlier RC issues have been successful. No further bug reports, so > 1.9.0 will probably be released on Monday.


Dominic Cleal
Red Hat Engineering

> >> Foreman 1.10
> >> ============
> >> Foreman 110 Schedule - Foreman
> >>
> >> Based on a mid August release of 1.9.0, I set the anticipated 1.10.0
> >> date to the start of December. Branching will be mid-October, so we are
> >> 1.5 months into the 3.5 month development period.
> >
> > I was wondering what folks think about this pace. Is this too fast, too
> > slow? Are folks skipping version upgrades to lower amount of production
> > upgrades?
>
> The data from the survey early this year was pleasantly surprising:
> http://blog.theforeman.org/2015/02/foreman-community-survey-results.html
>
> "1.7.x was around for about one month and a half at the time the survey
> was released. We support the two latest minor versions (1.7 and 1.6 in
> this case) so 86% of the installations are officially supported"
>
> Though it's likely IMHO that the survey respondents weren't
> representative of the entire population.
>
> I generally think it's a pretty aggressive schedule for a large piece of
> software, but one that is working OK. I think we have a reasonable
> balance between the sizes of the releases, the effort going into RCs and
> maintenance.
>
> Moving to some sort of smaller (to 1/3 or 1/4 of the size), monthly
> release schedule is quite tempting to me too. In particular, I'd like
> to reduce the number of release candidates as that's a lot of extra work.

Less than 3 months makes it harder for any substantial plugin. The way
it is now, when you all merge a change that affects foreman_salt, I'll
get notified by the weekly Jenkins test (hopefuly) and I know I can fix
it somewhat at my leisure in time for the RC's.

Then I spin up the first RC and test the plugin to see if the tests
missed something, and fix it if needed (e.g. they didn't catch select2
broke the JS).

That pace works really well for me to maintain a plugin on the side. I
feel like monthly releases would put an extra time pressure and I'd
struggle to keep up.

If Foreman adopted monthly releases, with a long-term supported release
(6-12 month), then that would be ok and I could target LTS, but I'm not
sure that'd solve anything.

Another thought, why are the RC's so much work? Is it possible to
automate more? I know Katello is slightly different, but the release
process for 2.3 was not really ideal, and I spent a lot of time trying
to figure out all these manual steps. I know Eric is working on
more automation, but this again seems to be a case where the two
projects could benefit from efficiencies of working together.

Why can't Jenkins create the first RC including branch structure, Koji
tags, etc for Katello and Foreman with one button?

··· On Wed, Aug 12, 2015 at 04:08:32PM +0100, Dominic Cleal wrote: > On 12/08/15 15:53, Lukas Zapletal wrote:


Dominic Cleal
Red Hat Engineering


You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Best Regards,

Stephen Benjamin
Red Hat Engineering

The data from the survey early this year was pleasantly surprising:

"1.7.x was around for about one month and a half at the time the survey
was released. We support the two latest minor versions (1.7 and 1.6 in
this case) so 86% of the installations are officially supported"

Though it's likely IMHO that the survey respondents weren't
representative of the entire population.

I generally think it's a pretty aggressive schedule for a large piece of
software, but one that is working OK. I think we have a reasonable
balance between the sizes of the releases, the effort going into RCs and
maintenance.

Moving to some sort of smaller (to 1/3 or 1/4 of the size), monthly
release schedule is quite tempting to me too. In particular, I'd like
to reduce the number of release candidates as that's a lot of extra work.

··· On 12/08/15 15:53, Lukas Zapletal wrote: >> Foreman 1.10 >> ============ >> http://projects.theforeman.org/projects/foreman/wiki/Foreman_110_Schedule >> >> Based on a mid August release of 1.9.0, I set the anticipated 1.10.0 >> date to the start of December. Branching will be mid-October, so we are >> 1.5 months into the 3.5 month development period. > > I was wondering what folks think about this pace. Is this too fast, too > slow? Are folks skipping version upgrades to lower amount of production > upgrades?


Dominic Cleal
Red Hat Engineering

>>>> Foreman 1.10
>>>> ============
>>>> Foreman 110 Schedule - Foreman
>>>>
>>>> Based on a mid August release of 1.9.0, I set the anticipated 1.10.0
>>>> date to the start of December. Branching will be mid-October, so we are
>>>> 1.5 months into the 3.5 month development period.
>>>
>>> I was wondering what folks think about this pace. Is this too fast, too
>>> slow? Are folks skipping version upgrades to lower amount of production
>>> upgrades?
>>
>> The data from the survey early this year was pleasantly surprising:
>> http://blog.theforeman.org/2015/02/foreman-community-survey-results.html
>>
>> "1.7.x was around for about one month and a half at the time the survey
>> was released. We support the two latest minor versions (1.7 and 1.6 in
>> this case) so 86% of the installations are officially supported"
>>
>> Though it's likely IMHO that the survey respondents weren't
>> representative of the entire population.
>>
>> I generally think it's a pretty aggressive schedule for a large piece of
>> software, but one that is working OK. I think we have a reasonable
>> balance between the sizes of the releases, the effort going into RCs and
>> maintenance.
>>
>> Moving to some sort of smaller (to 1/3 or 1/4 of the size), monthly
>> release schedule is quite tempting to me too. In particular, I'd like
>> to reduce the number of release candidates as that's a lot of extra work.
>
> Less than 3 months makes it harder for any substantial plugin. The way
> it is now, when you all merge a change that affects foreman_salt, I'll
> get notified by the weekly Jenkins test (hopefuly) and I know I can fix
> it somewhat at my leisure in time for the RC's.
>
> Then I spin up the first RC and test the plugin to see if the tests
> missed something, and fix it if needed (e.g. they didn't catch select2
> broke the JS).
>
> That pace works really well for me to maintain a plugin on the side. I
> feel like monthly releases would put an extra time pressure and I'd
> struggle to keep up.
>
> If Foreman adopted monthly releases, with a long-term supported release
> (6-12 month), then that would be ok and I could target LTS, but I'm not
> sure that'd solve anything.

Sure, you hit the nail on the head with one of the potential problems.
Sorry, I hadn't intended it as a serious proposal.

I would however like to avoid making the development period much longer,
as the releases increase in size and risk.

> Another thought, why are the RC's so much work? Is it possible to
> automate more? I know Katello is slightly different, but the release
> process for 2.3 was not really ideal, and I spent a lot of time trying
> to figure out all these manual steps. I know Eric is working on
> more automation, but this again seems to be a case where the two
> projects could benefit from efficiencies of working together.
>
> Why can't Jenkins create the first RC including branch structure, Koji
> tags, etc for Katello and Foreman with one button?

Branching happens 3-4 times a year and can be spread over a couple of
weeks as different subprojects become ready, so isn't something that I
think warrants full automation now, especially in Jenkins. It also
changes slightly on every release, which would make it harder to
maintain (steps are kept on Release_Process today). Some of it's
scripted, but generally it touches on a load of different things.

RCs are simply another minor release, it follows the same process.
During one major release, there will be around 6-9 RC+minor releases, so
cutting down that number saves time. Each RC/minor release takes up to
half a day's work, including cherry picking, tagging, running the
automation, updating packaging, verifying the automation, writing
releases notes etc.

I automated a lot of the packaging, test and release process a while
ago, and will continue to work on optimising bits of it as and when I can.

I'd like to continually build stable branches in the same way we do with
nightlies, but that requires some thought about how to do versioning and
tagging.

··· On 12/08/15 18:25, Stephen Benjamin wrote: > On Wed, Aug 12, 2015 at 04:08:32PM +0100, Dominic Cleal wrote: >> On 12/08/15 15:53, Lukas Zapletal wrote:


Dominic Cleal
Red Hat Engineering

> If Foreman adopted monthly releases, with a long-term supported release
> (6-12 month), then that would be ok and I could target LTS, but I'm not
> sure that'd solve anything.

I was actually thinking about one stable release per 5-6 months and some
kind of "stabilized nightly builds" every two weeks. But it looks like
only 2 per cent of our users do run nightlies.

··· -- Later, Lukas #lzap Zapletal

Personally, I'd like to shorten releases by automating the process as much
as possible. I'd love to be able to for the most part push a button to
release one or two commits at any point to the stable release to fix issues
encountered by users. The more often we release and the faster we release,
the easier it becomes and the more we are able to detect breakages (and
quickly react to them).

I like the idea of 'stable' nightlies, but I don't feel confident in our
system test suites verbosity at present to produce stable nightlies. The
Robotello discussion in the other thread, due to the breadth of the tests,
could be that which provides enough automated testing to feel confident in
stable nightlies.

Plugins breaking is a solid point and leaves me asking - why do plugins
break? Is it because Ruby allows plugins to modify or extend any code a
plugin wants to and we are essentially supporting this "API" ? Would a
defined, supported and versioned plugin API that provides the ability to
make the kind of changes plugins do today through defined extension points
alleviate this pain? Are there aspects that we should consider breaking out
into other 'modules' so that their cadence can go faster than other parts?

··· On Thu, Aug 13, 2015 at 5:25 AM, Dominic Cleal wrote:

On 13/08/15 10:23, Dominic Cleal wrote:

On 13/08/15 10:06, Lukas Zapletal wrote:

If Foreman adopted monthly releases, with a long-term supported release
(6-12 month), then that would be ok and I could target LTS, but I’m not
sure that’d solve anything.

I was actually thinking about one stable release per 5-6 months and some
kind of “stabilized nightly builds” every two weeks. But it looks like
only 2 per cent of our users do run nightlies.

I’d have a few concerns based on current behaviours:

We did a release (around 1.4?) which contained about 5-6 months worth of
changes, and it was one of the more difficult to stabilise, as you’d
expect.

We’d need to have a much bigger commitment from every developer to fix
regressions as there’d be many more, while currently it’s only a small
number of regular contributors who help with bug reports in the weeks
during and after releases.

The support period would be far longer, so backporting becomes
increasingly harder. Many changes can’t be backported safely, so it’s
longer before some fixes can be released - today many complex bug fixes
are only released in quarterly releases. The answer to that might be a
semver major/minor/patch style release system, having major releases
every 6 months instead?

I’m not sure what to think of the stabilised nightlies suggestion. I
think given the effort that might take, choose between doing that and
providing regular updates to a major release. We’d probably have
relatively poor compatibility with plugins on those releases, making it
less attractive to users.

I meant to also say, thanks for bringing the subject up. I’m happy to
have some discussion around it! The schedule has become defacto, but
doesn’t mean it’s the best solution.


Dominic Cleal
Red Hat Engineering


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University

I'd have a few concerns based on current behaviours:

We did a release (around 1.4?) which contained about 5-6 months worth of
changes, and it was one of the more difficult to stabilise, as you'd expect.

We'd need to have a much bigger commitment from every developer to fix
regressions as there'd be many more, while currently it's only a small
number of regular contributors who help with bug reports in the weeks
during and after releases.

The support period would be far longer, so backporting becomes
increasingly harder. Many changes can't be backported safely, so it's
longer before some fixes can be released - today many complex bug fixes
are only released in quarterly releases. The answer to that might be a
semver major/minor/patch style release system, having major releases
every 6 months instead?

I'm not sure what to think of the stabilised nightlies suggestion. I
think given the effort that might take, choose between doing that and
providing regular updates to a major release. We'd probably have
relatively poor compatibility with plugins on those releases, making it
less attractive to users.

··· On 13/08/15 10:06, Lukas Zapletal wrote: >> If Foreman adopted monthly releases, with a long-term supported release >> (6-12 month), then that would be ok and I could target LTS, but I'm not >> sure that'd solve anything. > > I was actually thinking about one stable release per 5-6 months and some > kind of "stabilized nightly builds" every two weeks. But it looks like > only 2 per cent of our users do run nightlies.


Dominic Cleal
Red Hat Engineering

I meant to also say, thanks for bringing the subject up. I'm happy to
have some discussion around it! The schedule has become defacto, but
doesn't mean it's the best solution.

··· On 13/08/15 10:23, Dominic Cleal wrote: > On 13/08/15 10:06, Lukas Zapletal wrote: >>> If Foreman adopted monthly releases, with a long-term supported release >>> (6-12 month), then that would be ok and I could target LTS, but I'm not >>> sure that'd solve anything. >> >> I was actually thinking about one stable release per 5-6 months and some >> kind of "stabilized nightly builds" every two weeks. But it looks like >> only 2 per cent of our users do run nightlies. > > I'd have a few concerns based on current behaviours: > > We did a release (around 1.4?) which contained about 5-6 months worth of > changes, and it was one of the more difficult to stabilise, as you'd expect. > > We'd need to have a much bigger commitment from every developer to fix > regressions as there'd be many more, while currently it's only a small > number of regular contributors who help with bug reports in the weeks > during and after releases. > > The support period would be far longer, so backporting becomes > increasingly harder. Many changes can't be backported safely, so it's > longer before some fixes can be released - today many complex bug fixes > are only released in quarterly releases. The answer to that might be a > semver major/minor/patch style release system, having major releases > every 6 months instead? > > I'm not sure what to think of the stabilised nightlies suggestion. I > think given the effort that might take, choose between doing that and > providing regular updates to a major release. We'd probably have > relatively poor compatibility with plugins on those releases, making it > less attractive to users.


Dominic Cleal
Red Hat Engineering

> Plugins breaking is a solid point and leaves me asking - why do plugins
> break? Is it because Ruby allows plugins to modify or extend any code a
> plugin wants to and we are essentially supporting this "API" ?

Yes, so even the mildest refactoring can easily break some fragile
method chaining, for instance. A few plugins also heavily reuse
partials and JavaScript in the UI, so are very fragile.

> Would a
> defined, supported and versioned plugin API that provides the ability to
> make the kind of changes plugins do today through defined extension
> points alleviate this pain?

Indeed, and this continues to improve I think. Some focus on examining
existing plugins and refactoring Foreman to provide defined extension
points for common tasks would be useful.

··· On 13/08/15 13:34, Eric D Helms wrote:


Dominic Cleal
Red Hat Engineering