Release process & permissions

Development
#1

Hi devs,

After a few releases, and now that I'm trying to help someone else to
take over in case it's needed, I found a roadblock.

Whoever is doing the release, needs to have many permissions.

Otherwise, it doesn't make much sense for a person to take over release
responsibilities. For example, if Ondrej has to do 1.15.5, he would need
the following permissions (see at the end of the email).

Of course there are alternatives:

1 is to have the release nanny be supervised by people who have 'earned'
these permissions. This is a bad idea because some of the tasks just
cannot be 'supervised'. The nanny would have to ask someone to tag
repositories, modify jenkins jobs, upload GPG signatures, post to the
mailing list, tag new builds in Koji…

2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and
make it a real pipeline from 0 to release completed. At this moment,
releases that are not the first RC1 are mostly automated by
https://github.com/dlobatog/foreman_release and
https://github.com/theforeman/tool_belt.

My proposal is to go forward with 2. Give Jenkins permissions to do all
of the actions needed, and whoever is the release nanny, ideally only
has to make sure all of the steps are moving forward. If something
breaks, figure out how to fix it for the next release.

This would mean making a few extra jobs before and after the current
release pipeline. In my opinion, it's the way to go to ensure anyone can
take over this responsibility.

At this moment, we are in a situation where only people who
mostly have permissions everywhere can successfully do a release without
asking many people for favors.

Personally if we complete this, I see it as a big win as it would dwarf
our bus factor for release managers & allow us to release at any pace we
desire (right now it's slow because we can't truly release things from
one day to the next due to the work involved).

Thoughts?

Here's the list of permissions:

··· ----------------

Github:

  • Push in foreman, foreman-selinux, foreman-installer,
    smart-proxy, foreman-infra, foreman-packaging

Transifex -

  • Allow to change the auto-update URL to point to latest -stable
    branch

Redmine -

  • Create new “Found in Release” version

Jenkins -

  • Modify jobs
  • Run jobs

Koji -

  • Create tags
  • SSH access to update the mash scripts
  • Create packages
  • Tag builds

Repository servers

Announcements -

  • Post to foreman-announce
  • Merge access in theforeman.org
  • Change IRC message
  • Publish in Twitter, G+

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

#2

… isn‘t continuous delivery old these days? Why aren‘t we doing it?
I vote in favor of automating this. This ensures predictable results and hopefully makes this process easier. To be honest: The current process scares me.
Same for plugin releases. They also are way too manual right now.

Timo

··· > On 27. Sep 2017, at 16:46, Daniel Lobato Garcia wrote: > > Hi devs, > > After a few releases, and now that I'm trying to help someone else to > take over in case it's needed, I found a roadblock. > > Whoever is doing the release, needs to have **many** permissions. > > Otherwise, it doesn't make much sense for a person to take over release > responsibilities. For example, if Ondrej has to do 1.15.5, he would need > the following permissions (see at the end of the email). > > Of course there are alternatives: > > 1 is to have the release nanny be supervised by people who have 'earned' > these permissions. This is a bad idea because some of the tasks just > cannot be 'supervised'. The nanny would have to ask someone to tag > repositories, modify jenkins jobs, upload GPG signatures, post to the > mailing list, tag new builds in Koji... > > 2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and > make it a real pipeline from 0 to release completed. At this moment, > releases that are not the first RC1 are mostly automated by > https://github.com/dlobatog/foreman_release and > https://github.com/theforeman/tool_belt. > > My proposal is to go forward with 2. Give Jenkins permissions to do all > of the actions needed, and whoever is the release nanny, ideally only > has to make sure all of the steps are moving forward. If something > breaks, figure out how to fix it for the next release. > > This would mean making a few extra jobs before and after the current > release pipeline. In my opinion, it's the way to go to ensure anyone can > take over this responsibility. > > At this moment, we are in a situation where only people who > mostly have permissions everywhere can successfully do a release without > asking many people for favors. > > Personally if we complete this, I see it as a big win as it would dwarf > our bus factor for release managers & allow us to release at any pace we > desire (right now it's slow because we can't truly release things from > one day to the next due to the work involved). > > Thoughts? > > Here's the list of permissions: > > ---------------- > > Github: > - Push in foreman, foreman-selinux, foreman-installer, > smart-proxy, foreman-infra, foreman-packaging > > Transifex - > - Allow to change the auto-update URL to point to latest -stable > branch > > Redmine - > - Create new "Found in Release" version > > Jenkins - > - Modify jobs > - Run jobs > > Koji - > - Create tags > - SSH access to update the mash scripts > - Create packages > - Tag builds > > Repository servers > - ssh in deb.theforeman.org > - ssh in yum.theforeman.org > > Announcements - > - Post to foreman-announce > - Merge access in theforeman.org > - Change IRC message > - Publish in Twitter, G+ > > --------------- > > -- > Daniel Lobato Garcia > > @dLobatog > blog.daniellobato.me > daniellobato.me > > GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 > Keybase: https://keybase.io/elobato > > -- > You received this message because you are subscribed to the Google Groups "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.
#3

Hello

I like 2. At the same time, if it takes more then few weeks, I'd say let's
just give the release person all permission that are required. If we agree
that the person should do the release (in the worst case, we can do the
nomination process), we should make it as easy as possible for them to do the
job. I don't think anyone would break something on purpose. People tend to ask
more skilled people when they are uncertain.

Thanks for the write up!

··· -- Marek

On středa 27. září 2017 16:46:34 CEST Daniel Lobato Garcia wrote:

Hi devs,

After a few releases, and now that I’m trying to help someone else to
take over in case it’s needed, I found a roadblock.

Whoever is doing the release, needs to have many permissions.

Otherwise, it doesn’t make much sense for a person to take over release
responsibilities. For example, if Ondrej has to do 1.15.5, he would need
the following permissions (see at the end of the email).

Of course there are alternatives:

1 is to have the release nanny be supervised by people who have 'earned’
these permissions. This is a bad idea because some of the tasks just
cannot be ‘supervised’. The nanny would have to ask someone to tag
repositories, modify jenkins jobs, upload GPG signatures, post to the
mailing list, tag new builds in Koji…

2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and
make it a real pipeline from 0 to release completed. At this moment,
releases that are not the first RC1 are mostly automated by
https://github.com/dlobatog/foreman_release and
https://github.com/theforeman/tool_belt.

My proposal is to go forward with 2. Give Jenkins permissions to do all
of the actions needed, and whoever is the release nanny, ideally only
has to make sure all of the steps are moving forward. If something
breaks, figure out how to fix it for the next release.

This would mean making a few extra jobs before and after the current
release pipeline. In my opinion, it’s the way to go to ensure anyone can
take over this responsibility.

At this moment, we are in a situation where only people who
mostly have permissions everywhere can successfully do a release without
asking many people for favors.

Personally if we complete this, I see it as a big win as it would dwarf
our bus factor for release managers & allow us to release at any pace we
desire (right now it’s slow because we can’t truly release things from
one day to the next due to the work involved).

Thoughts?

Here’s the list of permissions:

Github:

  • Push in foreman, foreman-selinux, foreman-installer,
    smart-proxy, foreman-infra, foreman-packaging

Transifex -

  • Allow to change the auto-update URL to point to latest -stable
    branch

Redmine -

  • Create new “Found in Release” version

Jenkins -

  • Modify jobs
  • Run jobs

Koji -

  • Create tags
  • SSH access to update the mash scripts
  • Create packages
  • Tag builds

Repository servers

Announcements -

  • Post to foreman-announce
  • Merge access in theforeman.org
  • Change IRC message
  • Publish in Twitter, G+

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

#4

There are two considerations that I'd like to put forth as we think about
this:

  1. Instead of adding jobs, we re-think and re-write the release job in
    Pipeline syntax similar to my starter here –
    https://github.com/theforeman/foreman-infra/pull/323
  2. We don't automate all the things as there are some tasks that should be
    done by a human. The tedious, repetitive bits we should automate. The
    aspects that require some human foresight, approval or double checking of
    we should either require a releaser to "yes" to a job over or to perform
    semi-automated in that the user uses tooling but ultimately has input. For
    example, cherry picking should be 90% automated but 10% human input to
    ensure nothing seems off since our issue-to-change is not flawless.

While we have some automation already in place, from my experience I would
recommend one of the following approaches.

  1. create a flow chart of every action that has to happen using something
    like plantuml with parallel actions where possible
  2. Create a new release job, starting either from the beginning or the end
    of the process and add each next step to it

Eric

··· On Wed, Sep 27, 2017 at 10:46 AM, Daniel Lobato Garcia wrote:

Hi devs,

After a few releases, and now that I’m trying to help someone else to
take over in case it’s needed, I found a roadblock.

Whoever is doing the release, needs to have many permissions.

Otherwise, it doesn’t make much sense for a person to take over release
responsibilities. For example, if Ondrej has to do 1.15.5, he would need
the following permissions (see at the end of the email).

Of course there are alternatives:

1 is to have the release nanny be supervised by people who have ‘earned’
these permissions. This is a bad idea because some of the tasks just
cannot be ‘supervised’. The nanny would have to ask someone to tag
repositories, modify jenkins jobs, upload GPG signatures, post to the
mailing list, tag new builds in Koji…

2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and
make it a real pipeline from 0 to release completed. At this moment,
releases that are not the first RC1 are mostly automated by
https://github.com/dlobatog/foreman_release and
https://github.com/theforeman/tool_belt.

My proposal is to go forward with 2. Give Jenkins permissions to do all
of the actions needed, and whoever is the release nanny, ideally only
has to make sure all of the steps are moving forward. If something
breaks, figure out how to fix it for the next release.

This would mean making a few extra jobs before and after the current
release pipeline. In my opinion, it’s the way to go to ensure anyone can
take over this responsibility.

At this moment, we are in a situation where only people who
mostly have permissions everywhere can successfully do a release without
asking many people for favors.

Personally if we complete this, I see it as a big win as it would dwarf
our bus factor for release managers & allow us to release at any pace we
desire (right now it’s slow because we can’t truly release things from
one day to the next due to the work involved).

Thoughts?

Here’s the list of permissions:

Github:

  • Push in foreman, foreman-selinux, foreman-installer,
    smart-proxy, foreman-infra, foreman-packaging

Transifex -

  • Allow to change the auto-update URL to point to latest -stable
    branch

Redmine -

  • Create new “Found in Release” version

Jenkins -

  • Modify jobs
  • Run jobs

Koji -

  • Create tags
  • SSH access to update the mash scripts
  • Create packages
  • Tag builds

Repository servers

Announcements -

  • Post to foreman-announce
  • Merge access in theforeman.org
  • Change IRC message
  • Publish in Twitter, G+

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering

#5

For foreman-docker I had a process:

https://github.com/theforeman/foreman-docker/blob/master/release/RELEASE.md
https://github.com/theforeman/foreman-docker/blob/master/release/rollout-release

which I planned to implement in all theforeman org plugins ideally.
It didn't happen but it's a similar thing.

··· On Wednesday, September 27, 2017 at 4:56:51 PM UTC+2, Timo Goebel wrote: > > ... isn‘t continuous delivery old these days? Why aren‘t we doing it? > I vote in favor of automating this. This ensures predictable results and > hopefully makes this process easier. To be honest: The current process > scares me. > Same for plugin releases. They also are way too manual right now. > > Timo > > > On 27. Sep 2017, at 16:46, Daniel Lobato Garcia > wrote: > > > > Hi devs, > > > > After a few releases, and now that I'm trying to help someone else to > > take over in case it's needed, I found a roadblock. > > > > Whoever is doing the release, needs to have **many** permissions. > > > > Otherwise, it doesn't make much sense for a person to take over release > > responsibilities. For example, if Ondrej has to do 1.15.5, he would need > > the following permissions (see at the end of the email). > > > > Of course there are alternatives: > > > > 1 is to have the release nanny be supervised by people who have 'earned' > > these permissions. This is a bad idea because some of the tasks just > > cannot be 'supervised'. The nanny would have to ask someone to tag > > repositories, modify jenkins jobs, upload GPG signatures, post to the > > mailing list, tag new builds in Koji... > > > > 2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and > > make it a real pipeline from 0 to release completed. At this moment, > > releases that are not the first RC1 are mostly automated by > > https://github.com/dlobatog/foreman_release and > > https://github.com/theforeman/tool_belt. > > > > My proposal is to go forward with 2. Give Jenkins permissions to do all > > of the actions needed, and whoever is the release nanny, ideally only > > has to make sure all of the steps are moving forward. If something > > breaks, figure out how to fix it for the next release. > > > > This would mean making a few extra jobs before and after the current > > release pipeline. In my opinion, it's the way to go to ensure anyone can > > take over this responsibility. > > > > At this moment, we are in a situation where only people who > > mostly have permissions everywhere can successfully do a release without > > asking many people for favors. > > > > Personally if we complete this, I see it as a big win as it would dwarf > > our bus factor for release managers & allow us to release at any pace we > > desire (right now it's slow because we can't truly release things from > > one day to the next due to the work involved). > > > > Thoughts? > > > > Here's the list of permissions: > > > > ---------------- > > > > Github: > > - Push in foreman, foreman-selinux, foreman-installer, > > smart-proxy, foreman-infra, foreman-packaging > > > > Transifex - > > - Allow to change the auto-update URL to point to latest -stable > > branch > > > > Redmine - > > - Create new "Found in Release" version > > > > Jenkins - > > - Modify jobs > > - Run jobs > > > > Koji - > > - Create tags > > - SSH access to update the mash scripts > > - Create packages > > - Tag builds > > > > Repository servers > > - ssh in deb.theforeman.org > > - ssh in yum.theforeman.org > > > > Announcements - > > - Post to foreman-announce > > - Merge access in theforeman.org > > - Change IRC message > > - Publish in Twitter, G+ > > > > --------------- > > > > -- > > Daniel Lobato Garcia > > > > @dLobatog > > blog.daniellobato.me > > daniellobato.me > > > > GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 > > Keybase: https://keybase.io/elobato > > > > -- > > You received this message because you are subscribed to the Google > Groups "foreman-dev" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to foreman-dev...@googlegroups.com . > > For more options, visit https://groups.google.com/d/optout. > >
#6

> >
> > 2 is to extend http://ci.theforeman.org/view/Release%20pipeline/
> > and make it a real pipeline from 0 to release completed. At this
> > moment, releases that are not the first RC1 are mostly automated by
> > https://github.com/dlobatog/foreman_release and
> > https://github.com/theforeman/tool_belt.
> >
> > My proposal is to go forward with 2. Give Jenkins permissions to do
> > all of the actions needed, and whoever is the release nanny,
> > ideally only has to make sure all of the steps are moving forward.
> > If something breaks, figure out how to fix it for the next release.

I agree, lets do it. I think I have permissions for most things,
excepting Koji, so I can help. Some thoughts inline:

Github:
> - Push in foreman, foreman-selinux, foreman-installer,
> smart-proxy, foreman-infra, foreman-packaging

Fine, any issues can easily be reverted.

> Transifex -
> - Allow to change the auto-update URL to point to latest -stable
> branch

I assume the Transifex API can handle this?

> Redmine -
> - Create new "Found in Release" version

API should be able to handle this, or we write a trivial plugin to
expose it

> Jenkins -
> - Modify jobs
> - Run jobs

Allow Jenkins to modify iteself? EEEK, Skynet :slight_smile:

Seriously though, what's needed here? Wouldn't the required versions be
input variables to the pipeline anyway?

> Koji -
> - Create tags
> - SSH access to update the mash scripts
> - Create packages
> - Tag builds

Not my area…

> Repository servers
> - ssh in deb.theforeman.org
> - ssh in yum.theforeman.org

This can easily be done, we already have limited-use SSH keys for
Jenkins in the deploy_web,etc jobs. Best approach is a script for what
we need that can be called from Jenkins.

> Announcements -
> - Post to foreman-announce
> - Merge access in theforeman.org
> - Change IRC message
> - Publish in Twitter, G+

G+ isn't so hot for us these days, I don't do much there. Which is just
as well, because setting up Google OAuth stuff is a giant pain in the
ass. The rest seems sane, but does re-open the discussion about
foreman-announce - I think more than just core releases should be put
there, and if we have a pipeline for it, then… good?

··· On 27. Sep 2017, at 16:46, Daniel Lobato Garcia > > wrote:

On Wed, 2017-09-27 at 16:56 +0200, Timo Goebel wrote:

Same for plugin releases. They also are way too manual right now.

+1 - if we can get a "jenkins@theforeman.org" author added to every
gem, then we can automate this and prevent missing author issues…

Greg

#7

While I agree we should automate a lot, I agree with Eric. Doing a
Foreman release should involve a human to keep the end-to-end quality.
Just giving permissions is wrong.

For example in Foreman 1.15.5 the installer was tagged and released
before I could push in my changes. That communication failure might have
been partly my fault but IMHO the job of a release manager is to
communicate with all the maintainers if everything is in place for a
release.

Now you state it as a problem that you need a lot of people, but you
need to communicate with them anyway. Yes, the tagging and releasing
should be scripted but planning a release will require humans.

Like Eric I think we should start by documenting. Right now it's a huge
black box for most people, let's start there.

··· On Wed, Sep 27, 2017 at 09:19:10PM -0400, Eric D Helms wrote: >There are two considerations that I'd like to put forth as we think about >this: > > 1) Instead of adding jobs, we re-think and re-write the release job in >Pipeline syntax similar to my starter here -- >https://github.com/theforeman/foreman-infra/pull/323 > 2) We don't automate all the things as there are some tasks that should be >done by a human. The tedious, repetitive bits we should automate. The >aspects that require some human foresight, approval or double checking of >we should either require a releaser to "yes" to a job over or to perform >semi-automated in that the user uses tooling but ultimately has input. For >example, cherry picking should be 90% automated but 10% human input to >ensure nothing seems off since our issue-to-change is not flawless. > >While we have some automation already in place, from my experience I would >recommend one of the following approaches. > > 1) create a flow chart of every action that has to happen using something >like plantuml with parallel actions where possible > 2) Create a new release job, starting either from the beginning or the end >of the process and add each next step to it > > >Eric > >On Wed, Sep 27, 2017 at 10:46 AM, Daniel Lobato Garcia >wrote: > >> Hi devs, >> >> After a few releases, and now that I'm trying to help someone else to >> take over in case it's needed, I found a roadblock. >> >> Whoever is doing the release, needs to have **many** permissions. >> >> Otherwise, it doesn't make much sense for a person to take over release >> responsibilities. For example, if Ondrej has to do 1.15.5, he would need >> the following permissions (see at the end of the email). >> >> Of course there are alternatives: >> >> 1 is to have the release nanny be supervised by people who have 'earned' >> these permissions. This is a bad idea because some of the tasks just >> cannot be 'supervised'. The nanny would have to ask someone to tag >> repositories, modify jenkins jobs, upload GPG signatures, post to the >> mailing list, tag new builds in Koji... >> >> 2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and >> make it a real pipeline from 0 to release completed. At this moment, >> releases that are not the first RC1 are mostly automated by >> https://github.com/dlobatog/foreman_release and >> https://github.com/theforeman/tool_belt. >> >> My proposal is to go forward with 2. Give Jenkins permissions to do all >> of the actions needed, and whoever is the release nanny, ideally only >> has to make sure all of the steps are moving forward. If something >> breaks, figure out how to fix it for the next release. >> >> This would mean making a few extra jobs before and after the current >> release pipeline. In my opinion, it's the way to go to ensure anyone can >> take over this responsibility. >> >> At this moment, we are in a situation where only people who >> mostly have permissions everywhere can successfully do a release without >> asking many people for favors. >> >> Personally if we complete this, I see it as a big win as it would dwarf >> our bus factor for release managers & allow us to release at any pace we >> desire (right now it's slow because we can't truly release things from >> one day to the next due to the work involved). >> >> Thoughts? >> >> Here's the list of permissions: >> >> ---------------- >> >> Github: >> - Push in foreman, foreman-selinux, foreman-installer, >> smart-proxy, foreman-infra, foreman-packaging >> >> Transifex - >> - Allow to change the auto-update URL to point to latest -stable >> branch >> >> Redmine - >> - Create new "Found in Release" version >> >> Jenkins - >> - Modify jobs >> - Run jobs >> >> Koji - >> - Create tags >> - SSH access to update the mash scripts >> - Create packages >> - Tag builds >> >> Repository servers >> - ssh in deb.theforeman.org >> - ssh in yum.theforeman.org >> >> Announcements - >> - Post to foreman-announce >> - Merge access in theforeman.org >> - Change IRC message >> - Publish in Twitter, G+ >> >> --------------- >> >> -- >> Daniel Lobato Garcia >> >> @dLobatog >> blog.daniellobato.me >> daniellobato.me >> >> GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 >> Keybase: https://keybase.io/elobato >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > >-- >Eric D. Helms >Red Hat Engineering > >-- >You received this message because you are subscribed to the Google Groups "foreman-dev" group. >To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. >For more options, visit https://groups.google.com/d/optout.
#8

To release a gem to rubygems I'd recommend looking at how voxpupuli
implemented this using Travis1. The same can be done for puppet2.
That means you can push a tag to git and the release is there.

There are also tools that help you bump. For puppet there is
puppet-blacksmith3. How to do annotated tags is in my notes3. A
more generic script is bump2version5 and I'm sure there are similar
tools in the ruby world.

IMHO these tools should be suggested in the plugin templates.

··· On Wed, Sep 27, 2017 at 08:08:46AM -0700, Daniel Lobato wrote: >For foreman-docker I had a process: > > https://github.com/theforeman/foreman-docker/blob/master/release/RELEASE.md > https://github.com/theforeman/foreman-docker/blob/master/release/rollout-release > >which I planned to implement in all theforeman org plugins ideally. >It didn't happen but it's a similar thing. > >On Wednesday, September 27, 2017 at 4:56:51 PM UTC+2, Timo Goebel wrote: >> >> ... isn‘t continuous delivery old these days? Why aren‘t we doing it? >> I vote in favor of automating this. This ensures predictable results and >> hopefully makes this process easier. To be honest: The current process >> scares me. >> Same for plugin releases. They also are way too manual right now. >> >> Timo >> >> > On 27. Sep 2017, at 16:46, Daniel Lobato Garcia > > wrote: >> > >> > Hi devs, >> > >> > After a few releases, and now that I'm trying to help someone else to >> > take over in case it's needed, I found a roadblock. >> > >> > Whoever is doing the release, needs to have **many** permissions. >> > >> > Otherwise, it doesn't make much sense for a person to take over release >> > responsibilities. For example, if Ondrej has to do 1.15.5, he would need >> > the following permissions (see at the end of the email). >> > >> > Of course there are alternatives: >> > >> > 1 is to have the release nanny be supervised by people who have 'earned' >> > these permissions. This is a bad idea because some of the tasks just >> > cannot be 'supervised'. The nanny would have to ask someone to tag >> > repositories, modify jenkins jobs, upload GPG signatures, post to the >> > mailing list, tag new builds in Koji... >> > >> > 2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and >> > make it a real pipeline from 0 to release completed. At this moment, >> > releases that are not the first RC1 are mostly automated by >> > https://github.com/dlobatog/foreman_release and >> > https://github.com/theforeman/tool_belt. >> > >> > My proposal is to go forward with 2. Give Jenkins permissions to do all >> > of the actions needed, and whoever is the release nanny, ideally only >> > has to make sure all of the steps are moving forward. If something >> > breaks, figure out how to fix it for the next release. >> > >> > This would mean making a few extra jobs before and after the current >> > release pipeline. In my opinion, it's the way to go to ensure anyone can >> > take over this responsibility. >> > >> > At this moment, we are in a situation where only people who >> > mostly have permissions everywhere can successfully do a release without >> > asking many people for favors. >> > >> > Personally if we complete this, I see it as a big win as it would dwarf >> > our bus factor for release managers & allow us to release at any pace we >> > desire (right now it's slow because we can't truly release things from >> > one day to the next due to the work involved). >> > >> > Thoughts? >> > >> > Here's the list of permissions: >> > >> > ---------------- >> > >> > Github: >> > - Push in foreman, foreman-selinux, foreman-installer, >> > smart-proxy, foreman-infra, foreman-packaging >> > >> > Transifex - >> > - Allow to change the auto-update URL to point to latest -stable >> > branch >> > >> > Redmine - >> > - Create new "Found in Release" version >> > >> > Jenkins - >> > - Modify jobs >> > - Run jobs >> > >> > Koji - >> > - Create tags >> > - SSH access to update the mash scripts >> > - Create packages >> > - Tag builds >> > >> > Repository servers >> > - ssh in deb.theforeman.org >> > - ssh in yum.theforeman.org >> > >> > Announcements - >> > - Post to foreman-announce >> > - Merge access in theforeman.org >> > - Change IRC message >> > - Publish in Twitter, G+ >> > >> > --------------- >> > >> > -- >> > Daniel Lobato Garcia >> > >> > @dLobatog >> > blog.daniellobato.me >> > daniellobato.me >> > >> > GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 >> > Keybase: https://keybase.io/elobato >> > >> > -- >> > You received this message because you are subscribed to the Google >> Groups "foreman-dev" group. >> > To unsubscribe from this group and stop receiving emails from it, send >> an email to foreman-dev...@googlegroups.com . >> > For more options, visit https://groups.google.com/d/optout. >> >> > >-- >You received this message because you are subscribed to the Google Groups "foreman-dev" group. >To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. >For more options, visit https://groups.google.com/d/optout.
#9

Why not distribute the release process? Each component can have (probably
already does) multiple maintainers who are capable of doing most of the
leg-work. The role of the release nanny then becomes coordinating the
effort, making public announcements and such. Such an approach would help
avoid overly broad permissions too.

-d

··· On Thu, Sep 28, 2017 at 3:23 AM, Ewoud Kohl van Wijngaarden < ewoud@kohlvanwijngaarden.nl> wrote:

To release a gem to rubygems I’d recommend looking at how voxpupuli
implemented this using Travis1. The same can be done for puppet[2]. That
means you can push a tag to git and the release is there.

There are also tools that help you bump. For puppet there is
puppet-blacksmith[3]. How to do annotated tags is in my notes[3]. A more
generic script is bump2version[5] and I’m sure there are similar tools in
the ruby world.

IMHO these tools should be suggested in the plugin templates.

fc4111511fb9b6a239ffcb2a3/.travis.yml#L18-L27
[2]: https://github.com/voxpupuli/puppet-nginx/blob/fc363114e509f
439b51ea22723a996ffca8a107d/.travis.yml#L48-L58
[3]: https://github.com/voxpupuli/puppet-blacksmith
[4]: http://pad-katello.rhcloud.com/p/releasing-modules
[5]: https://github.com/c4urself/bump2version

On Wed, Sep 27, 2017 at 08:08:46AM -0700, Daniel Lobato wrote:

For foreman-docker I had a process:

https://github.com/theforeman/foreman-docker/blob/master/rel
ease/RELEASE.md
https://github.com/theforeman/foreman-docker/blob/master/rel
ease/rollout-release

which I planned to implement in all theforeman org plugins ideally.
It didn’t happen but it’s a similar thing.

On Wednesday, September 27, 2017 at 4:56:51 PM UTC+2, Timo Goebel wrote:

… isn‘t continuous delivery old these days? Why aren‘t we doing it?
I vote in favor of automating this. This ensures predictable results and
hopefully makes this process easier. To be honest: The current process
scares me.
Same for plugin releases. They also are way too manual right now.

Timo

On 27. Sep 2017, at 16:46, Daniel Lobato Garcia <elob...@gmail.com >>> >>> <javascript:>> wrote:

Hi devs,

After a few releases, and now that I’m trying to help someone else to
take over in case it’s needed, I found a roadblock.

Whoever is doing the release, needs to have many permissions.

Otherwise, it doesn’t make much sense for a person to take over release
responsibilities. For example, if Ondrej has to do 1.15.5, he would
need
the following permissions (see at the end of the email).

Of course there are alternatives:

1 is to have the release nanny be supervised by people who have
’earned’
these permissions. This is a bad idea because some of the tasks just
cannot be ‘supervised’. The nanny would have to ask someone to tag
repositories, modify jenkins jobs, upload GPG signatures, post to the
mailing list, tag new builds in Koji…

2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and
make it a real pipeline from 0 to release completed. At this moment,
releases that are not the first RC1 are mostly automated by
https://github.com/dlobatog/foreman_release and
https://github.com/theforeman/tool_belt.

My proposal is to go forward with 2. Give Jenkins permissions to do all
of the actions needed, and whoever is the release nanny, ideally only
has to make sure all of the steps are moving forward. If something
breaks, figure out how to fix it for the next release.

This would mean making a few extra jobs before and after the current
release pipeline. In my opinion, it’s the way to go to ensure anyone
can
take over this responsibility.

At this moment, we are in a situation where only people who
mostly have permissions everywhere can successfully do a release
without
asking many people for favors.

Personally if we complete this, I see it as a big win as it would dwarf
our bus factor for release managers & allow us to release at any pace
we
desire (right now it’s slow because we can’t truly release things from
one day to the next due to the work involved).

Thoughts?

Here’s the list of permissions:

Github:

  • Push in foreman, foreman-selinux, foreman-installer,
    smart-proxy, foreman-infra, foreman-packaging

Transifex -

  • Allow to change the auto-update URL to point to latest -stable
    branch

Redmine -

  • Create new “Found in Release” version

Jenkins -

  • Modify jobs
  • Run jobs

Koji -

  • Create tags
  • SSH access to update the mash scripts
  • Create packages
  • Tag builds

Repository servers

Announcements -

  • Post to foreman-announce
  • Merge access in theforeman.org
  • Change IRC message
  • Publish in Twitter, G+

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

#10

Related to Dmitri's point, and I've thrown this question out with Katello
releases at least every other release, do all the components that are
currently released "together" need to be so these days? That is to say, can
any of the "versioned together components" be released more independently
but within the version stream? For example, say we are approaching 1.17
release, that is a good time to cut puppet releases, installer and
potentially smart proxy but do they need to be released together? Does
Foreman 1.17.1 necessitate a 1.17.1 installer? Or reverse it, does an
installer update have to wait on a Foreman 1.17.1 release?

Katello used to do this practice with a lot of repositories (e.g.
katello-agent, katello-selinux) and were at times releasing projects "just
because" without any real changes. After moving away from that, to let
katello-agent, katello-selinux and hammer-cli-katello more independently
release the process got a lot simpler. Further, I'd argue that each project
got a bit more focused and more thought put into maintenance and release.

Food for thought and discussion.

E

··· On Thu, Sep 28, 2017 at 2:29 PM, Dmitri Dolguikh wrote:

Why not distribute the release process? Each component can have (probably
already does) multiple maintainers who are capable of doing most of the
leg-work. The role of the release nanny then becomes coordinating the
effort, making public announcements and such. Such an approach would help
avoid overly broad permissions too.

-d

On Thu, Sep 28, 2017 at 3:23 AM, Ewoud Kohl van Wijngaarden < > ewoud@kohlvanwijngaarden.nl> wrote:

To release a gem to rubygems I’d recommend looking at how voxpupuli
implemented this using Travis1. The same can be done for puppet[2]. That
means you can push a tag to git and the release is there.

There are also tools that help you bump. For puppet there is
puppet-blacksmith[3]. How to do annotated tags is in my notes[3]. A more
generic script is bump2version[5] and I’m sure there are similar tools in
the ruby world.

IMHO these tools should be suggested in the plugin templates.

fc4111511fb9b6a239ffcb2a3/.travis.yml#L18-L27
[2]: https://github.com/voxpupuli/puppet-nginx/blob/fc363114e509f
439b51ea22723a996ffca8a107d/.travis.yml#L48-L58
[3]: https://github.com/voxpupuli/puppet-blacksmith
[4]: http://pad-katello.rhcloud.com/p/releasing-modules
[5]: https://github.com/c4urself/bump2version

On Wed, Sep 27, 2017 at 08:08:46AM -0700, Daniel Lobato wrote:

For foreman-docker I had a process:

https://github.com/theforeman/foreman-docker/blob/master/rel
ease/RELEASE.md
https://github.com/theforeman/foreman-docker/blob/master/rel
ease/rollout-release

which I planned to implement in all theforeman org plugins ideally.
It didn’t happen but it’s a similar thing.

On Wednesday, September 27, 2017 at 4:56:51 PM UTC+2, Timo Goebel wrote:

… isn‘t continuous delivery old these days? Why aren‘t we doing it?
I vote in favor of automating this. This ensures predictable results and
hopefully makes this process easier. To be honest: The current process
scares me.
Same for plugin releases. They also are way too manual right now.

Timo

On 27. Sep 2017, at 16:46, Daniel Lobato Garcia <elob...@gmail.com >>>> >>>> <javascript:>> wrote:

Hi devs,

After a few releases, and now that I’m trying to help someone else to
take over in case it’s needed, I found a roadblock.

Whoever is doing the release, needs to have many permissions.

Otherwise, it doesn’t make much sense for a person to take over
release
responsibilities. For example, if Ondrej has to do 1.15.5, he would
need
the following permissions (see at the end of the email).

Of course there are alternatives:

1 is to have the release nanny be supervised by people who have
’earned’
these permissions. This is a bad idea because some of the tasks just
cannot be ‘supervised’. The nanny would have to ask someone to tag
repositories, modify jenkins jobs, upload GPG signatures, post to the
mailing list, tag new builds in Koji…

2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and
make it a real pipeline from 0 to release completed. At this moment,
releases that are not the first RC1 are mostly automated by
https://github.com/dlobatog/foreman_release and
https://github.com/theforeman/tool_belt.

My proposal is to go forward with 2. Give Jenkins permissions to do
all
of the actions needed, and whoever is the release nanny, ideally only
has to make sure all of the steps are moving forward. If something
breaks, figure out how to fix it for the next release.

This would mean making a few extra jobs before and after the current
release pipeline. In my opinion, it’s the way to go to ensure anyone
can
take over this responsibility.

At this moment, we are in a situation where only people who
mostly have permissions everywhere can successfully do a release
without
asking many people for favors.

Personally if we complete this, I see it as a big win as it would
dwarf
our bus factor for release managers & allow us to release at any pace
we
desire (right now it’s slow because we can’t truly release things from
one day to the next due to the work involved).

Thoughts?

Here’s the list of permissions:

Github:

  • Push in foreman, foreman-selinux, foreman-installer,
    smart-proxy, foreman-infra, foreman-packaging

Transifex -

  • Allow to change the auto-update URL to point to latest -stable
    branch

Redmine -

  • Create new “Found in Release” version

Jenkins -

  • Modify jobs
  • Run jobs

Koji -

  • Create tags
  • SSH access to update the mash scripts
  • Create packages
  • Tag builds

Repository servers

Announcements -

  • Post to foreman-announce
  • Merge access in theforeman.org
  • Change IRC message
  • Publish in Twitter, G+

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D
6DE30
Keybase: https://keybase.io/elobato

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google
Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send
an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering

#11

IMHO that's another (better?) way of phrasing the point I made about
relasing being about communication in another email.

··· On Thu, Sep 28, 2017 at 11:29:36AM -0700, Dmitri Dolguikh wrote: >Why not distribute the release process? Each component can have (probably >already does) multiple maintainers who are capable of doing most of the >leg-work. The role of the release nanny then becomes coordinating the >effort, making public announcements and such. Such an approach would help >avoid overly broad permissions too. > >-d > >On Thu, Sep 28, 2017 at 3:23 AM, Ewoud Kohl van Wijngaarden < >ewoud@kohlvanwijngaarden.nl> wrote: > >> To release a gem to rubygems I'd recommend looking at how voxpupuli >> implemented this using Travis[1]. The same can be done for puppet[2]. That >> means you can push a tag to git and the release is there. >> >> There are also tools that help you bump. For puppet there is >> puppet-blacksmith[3]. How to do annotated tags is in my notes[3]. A more >> generic script is bump2version[5] and I'm sure there are similar tools in >> the ruby world. >> >> IMHO these tools should be suggested in the plugin templates. >> >> [1]: https://github.com/voxpupuli/modulesync/blob/e53079030501756 >> fc4111511fb9b6a239ffcb2a3/.travis.yml#L18-L27 >> [2]: https://github.com/voxpupuli/puppet-nginx/blob/fc363114e509f >> 439b51ea22723a996ffca8a107d/.travis.yml#L48-L58 >> [3]: https://github.com/voxpupuli/puppet-blacksmith >> [4]: http://pad-katello.rhcloud.com/p/releasing-modules >> [5]: https://github.com/c4urself/bump2version >> >> On Wed, Sep 27, 2017 at 08:08:46AM -0700, Daniel Lobato wrote: >> >>> For foreman-docker I had a process: >>> >>> https://github.com/theforeman/foreman-docker/blob/master/rel >>> ease/RELEASE.md >>> https://github.com/theforeman/foreman-docker/blob/master/rel >>> ease/rollout-release >>> >>> which I planned to implement in all theforeman org plugins ideally. >>> It didn't happen but it's a similar thing. >>> >>> On Wednesday, September 27, 2017 at 4:56:51 PM UTC+2, Timo Goebel wrote: >>> >>>> >>>> ... isn‘t continuous delivery old these days? Why aren‘t we doing it? >>>> I vote in favor of automating this. This ensures predictable results and >>>> hopefully makes this process easier. To be honest: The current process >>>> scares me. >>>> Same for plugin releases. They also are way too manual right now. >>>> >>>> Timo >>>> >>>> > On 27. Sep 2017, at 16:46, Daniel Lobato Garcia >>> >>>> > wrote: >>>> > >>>> > Hi devs, >>>> > >>>> > After a few releases, and now that I'm trying to help someone else to >>>> > take over in case it's needed, I found a roadblock. >>>> > >>>> > Whoever is doing the release, needs to have **many** permissions. >>>> > >>>> > Otherwise, it doesn't make much sense for a person to take over release >>>> > responsibilities. For example, if Ondrej has to do 1.15.5, he would >>>> need >>>> > the following permissions (see at the end of the email). >>>> > >>>> > Of course there are alternatives: >>>> > >>>> > 1 is to have the release nanny be supervised by people who have >>>> 'earned' >>>> > these permissions. This is a bad idea because some of the tasks just >>>> > cannot be 'supervised'. The nanny would have to ask someone to tag >>>> > repositories, modify jenkins jobs, upload GPG signatures, post to the >>>> > mailing list, tag new builds in Koji... >>>> > >>>> > 2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and >>>> > make it a real pipeline from 0 to release completed. At this moment, >>>> > releases that are not the first RC1 are mostly automated by >>>> > https://github.com/dlobatog/foreman_release and >>>> > https://github.com/theforeman/tool_belt. >>>> > >>>> > My proposal is to go forward with 2. Give Jenkins permissions to do all >>>> > of the actions needed, and whoever is the release nanny, ideally only >>>> > has to make sure all of the steps are moving forward. If something >>>> > breaks, figure out how to fix it for the next release. >>>> > >>>> > This would mean making a few extra jobs before and after the current >>>> > release pipeline. In my opinion, it's the way to go to ensure anyone >>>> can >>>> > take over this responsibility. >>>> > >>>> > At this moment, we are in a situation where only people who >>>> > mostly have permissions everywhere can successfully do a release >>>> without >>>> > asking many people for favors. >>>> > >>>> > Personally if we complete this, I see it as a big win as it would dwarf >>>> > our bus factor for release managers & allow us to release at any pace >>>> we >>>> > desire (right now it's slow because we can't truly release things from >>>> > one day to the next due to the work involved). >>>> > >>>> > Thoughts? >>>> > >>>> > Here's the list of permissions: >>>> > >>>> > ---------------- >>>> > >>>> > Github: >>>> > - Push in foreman, foreman-selinux, foreman-installer, >>>> > smart-proxy, foreman-infra, foreman-packaging >>>> > >>>> > Transifex - >>>> > - Allow to change the auto-update URL to point to latest -stable >>>> > branch >>>> > >>>> > Redmine - >>>> > - Create new "Found in Release" version >>>> > >>>> > Jenkins - >>>> > - Modify jobs >>>> > - Run jobs >>>> > >>>> > Koji - >>>> > - Create tags >>>> > - SSH access to update the mash scripts >>>> > - Create packages >>>> > - Tag builds >>>> > >>>> > Repository servers >>>> > - ssh in deb.theforeman.org >>>> > - ssh in yum.theforeman.org >>>> > >>>> > Announcements - >>>> > - Post to foreman-announce >>>> > - Merge access in theforeman.org >>>> > - Change IRC message >>>> > - Publish in Twitter, G+ >>>> > >>>> > --------------- >>>> > >>>> > -- >>>> > Daniel Lobato Garcia >>>> > >>>> > @dLobatog >>>> > blog.daniellobato.me >>>> > daniellobato.me >>>> > >>>> > GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30 >>>> > Keybase: https://keybase.io/elobato >>>> > >>>> > -- >>>> > You received this message because you are subscribed to the Google >>>> Groups "foreman-dev" group. >>>> > To unsubscribe from this group and stop receiving emails from it, send >>>> an email to foreman-dev...@googlegroups.com . >>>> > For more options, visit https://groups.google.com/d/optout. >>>> >>>> >>>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "foreman-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to foreman-dev+unsubscribe@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > >-- >You received this message because you are subscribed to the Google Groups "foreman-dev" group. >To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. >For more options, visit https://groups.google.com/d/optout.
#12

> While I agree we should automate a lot, I agree with Eric. Doing a Foreman
> release should involve a human to keep the end-to-end quality. Just giving
> permissions is wrong.
>
> For example in Foreman 1.15.5 the installer was tagged and released before I
> could push in my changes. That communication failure might have been partly
> my fault but IMHO the job of a release manager is to communicate with all
> the maintainers if everything is in place for a release.
>
> Now you state it as a problem that you need a lot of people, but you need to
> communicate with them anyway. Yes, the tagging and releasing should be
> scripted but planning a release will require humans.
>
> Like Eric I think we should start by documenting. Right now it's a huge
> black box for most people, let's start there.

The process is pretty well documented,
http://projects.theforeman.org/projects/foreman/wiki/Release_Process
contains everything that has to be done step by step

··· On 09/28, Ewoud Kohl van Wijngaarden wrote:

On Wed, Sep 27, 2017 at 09:19:10PM -0400, Eric D Helms wrote:

There are two considerations that I’d like to put forth as we think about
this:

  1. Instead of adding jobs, we re-think and re-write the release job in
    Pipeline syntax similar to my starter here –
    https://github.com/theforeman/foreman-infra/pull/323
  2. We don’t automate all the things as there are some tasks that should be
    done by a human. The tedious, repetitive bits we should automate. The
    aspects that require some human foresight, approval or double checking of
    we should either require a releaser to “yes” to a job over or to perform
    semi-automated in that the user uses tooling but ultimately has input. For
    example, cherry picking should be 90% automated but 10% human input to
    ensure nothing seems off since our issue-to-change is not flawless.

While we have some automation already in place, from my experience I would
recommend one of the following approaches.

  1. create a flow chart of every action that has to happen using something
    like plantuml with parallel actions where possible
  2. Create a new release job, starting either from the beginning or the end
    of the process and add each next step to it

Eric

On Wed, Sep 27, 2017 at 10:46 AM, Daniel Lobato Garcia elobatocs@gmail.com > > wrote:

Hi devs,

After a few releases, and now that I’m trying to help someone else to
take over in case it’s needed, I found a roadblock.

Whoever is doing the release, needs to have many permissions.

Otherwise, it doesn’t make much sense for a person to take over release
responsibilities. For example, if Ondrej has to do 1.15.5, he would need
the following permissions (see at the end of the email).

Of course there are alternatives:

1 is to have the release nanny be supervised by people who have 'earned’
these permissions. This is a bad idea because some of the tasks just
cannot be ‘supervised’. The nanny would have to ask someone to tag
repositories, modify jenkins jobs, upload GPG signatures, post to the
mailing list, tag new builds in Koji…

2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and
make it a real pipeline from 0 to release completed. At this moment,
releases that are not the first RC1 are mostly automated by
https://github.com/dlobatog/foreman_release and
https://github.com/theforeman/tool_belt.

My proposal is to go forward with 2. Give Jenkins permissions to do all
of the actions needed, and whoever is the release nanny, ideally only
has to make sure all of the steps are moving forward. If something
breaks, figure out how to fix it for the next release.

This would mean making a few extra jobs before and after the current
release pipeline. In my opinion, it’s the way to go to ensure anyone can
take over this responsibility.

At this moment, we are in a situation where only people who
mostly have permissions everywhere can successfully do a release without
asking many people for favors.

Personally if we complete this, I see it as a big win as it would dwarf
our bus factor for release managers & allow us to release at any pace we
desire (right now it’s slow because we can’t truly release things from
one day to the next due to the work involved).

Thoughts?

Here’s the list of permissions:

Github:

  • Push in foreman, foreman-selinux, foreman-installer,
    smart-proxy, foreman-infra, foreman-packaging

Transifex -

  • Allow to change the auto-update URL to point to latest -stable
    branch

Redmine -

  • Create new “Found in Release” version

Jenkins -

  • Modify jobs
  • Run jobs

Koji -

  • Create tags
  • SSH access to update the mash scripts
  • Create packages
  • Tag builds

Repository servers

Announcements -

  • Post to foreman-announce
  • Merge access in theforeman.org
  • Change IRC message
  • Publish in Twitter, G+

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Daniel Lobato Garcia

@dLobatog
blog.daniellobato.me
daniellobato.me

GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
Keybase: https://keybase.io/elobato

#13

+1 on not doing a release just for the sake of it. Sometimes it's needed
but most of the time it's not. See the differences between 1.15.0 and
1.15.4:

https://github.com/theforeman/foreman-installer/compare/1.15.0...1.15.4

··· On Thu, Sep 28, 2017 at 07:57:34PM -0400, Eric D Helms wrote: >Related to Dmitri's point, and I've thrown this question out with Katello >releases at least every other release, do all the components that are >currently released "together" need to be so these days? That is to say, can >any of the "versioned together components" be released more independently >but within the version stream? For example, say we are approaching 1.17 >release, that is a good time to cut puppet releases, installer and >potentially smart proxy but do they need to be released together? Does >Foreman 1.17.1 necessitate a 1.17.1 installer? Or reverse it, does an >installer update have to wait on a Foreman 1.17.1 release? > >Katello used to do this practice with a lot of repositories (e.g. >katello-agent, katello-selinux) and were at times releasing projects "just >because" without any real changes. After moving away from that, to let >katello-agent, katello-selinux and hammer-cli-katello more independently >release the process got a lot simpler. Further, I'd argue that each project >got a bit more focused and more thought put into maintenance and release. > >Food for thought and discussion. > >E > > >On Thu, Sep 28, 2017 at 2:29 PM, Dmitri Dolguikh >wrote: > >> Why not distribute the release process? Each component can have (probably >> already does) multiple maintainers who are capable of doing most of the >> leg-work. The role of the release nanny then becomes coordinating the >> effort, making public announcements and such. Such an approach would help >> avoid overly broad permissions too. >> >> -d >> >> On Thu, Sep 28, 2017 at 3:23 AM, Ewoud Kohl van Wijngaarden < >> ewoud@kohlvanwijngaarden.nl> wrote: >> >>> To release a gem to rubygems I'd recommend looking at how voxpupuli >>> implemented this using Travis[1]. The same can be done for puppet[2]. That >>> means you can push a tag to git and the release is there. >>> >>> There are also tools that help you bump. For puppet there is >>> puppet-blacksmith[3]. How to do annotated tags is in my notes[3]. A more >>> generic script is bump2version[5] and I'm sure there are similar tools in >>> the ruby world. >>> >>> IMHO these tools should be suggested in the plugin templates. >>> >>> [1]: https://github.com/voxpupuli/modulesync/blob/e53079030501756 >>> fc4111511fb9b6a239ffcb2a3/.travis.yml#L18-L27 >>> [2]: https://github.com/voxpupuli/puppet-nginx/blob/fc363114e509f >>> 439b51ea22723a996ffca8a107d/.travis.yml#L48-L58 >>> [3]: https://github.com/voxpupuli/puppet-blacksmith >>> [4]: http://pad-katello.rhcloud.com/p/releasing-modules >>> [5]: https://github.com/c4urself/bump2version >>> >>> On Wed, Sep 27, 2017 at 08:08:46AM -0700, Daniel Lobato wrote: >>> >>>> For foreman-docker I had a process: >>>> >>>> https://github.com/theforeman/foreman-docker/blob/master/rel >>>> ease/RELEASE.md >>>> https://github.com/theforeman/foreman-docker/blob/master/rel >>>> ease/rollout-release >>>> >>>> which I planned to implement in all theforeman org plugins ideally. >>>> It didn't happen but it's a similar thing. >>>> >>>> On Wednesday, September 27, 2017 at 4:56:51 PM UTC+2, Timo Goebel wrote: >>>> >>>>> >>>>> ... isn‘t continuous delivery old these days? Why aren‘t we doing it? >>>>> I vote in favor of automating this. This ensures predictable results and >>>>> hopefully makes this process easier. To be honest: The current process >>>>> scares me. >>>>> Same for plugin releases. They also are way too manual right now. >>>>> >>>>> Timo >>>>> >>>>> > On 27. Sep 2017, at 16:46, Daniel Lobato Garcia >>>> >>>>> > wrote: >>>>> > >>>>> > Hi devs, >>>>> > >>>>> > After a few releases, and now that I'm trying to help someone else to >>>>> > take over in case it's needed, I found a roadblock. >>>>> > >>>>> > Whoever is doing the release, needs to have **many** permissions. >>>>> > >>>>> > Otherwise, it doesn't make much sense for a person to take over >>>>> release >>>>> > responsibilities. For example, if Ondrej has to do 1.15.5, he would >>>>> need >>>>> > the following permissions (see at the end of the email). >>>>> > >>>>> > Of course there are alternatives: >>>>> > >>>>> > 1 is to have the release nanny be supervised by people who have >>>>> 'earned' >>>>> > these permissions. This is a bad idea because some of the tasks just >>>>> > cannot be 'supervised'. The nanny would have to ask someone to tag >>>>> > repositories, modify jenkins jobs, upload GPG signatures, post to the >>>>> > mailing list, tag new builds in Koji... >>>>> > >>>>> > 2 is to extend http://ci.theforeman.org/view/Release%20pipeline/ and >>>>> > make it a real pipeline from 0 to release completed. At this moment, >>>>> > releases that are not the first RC1 are mostly automated by >>>>> > https://github.com/dlobatog/foreman_release and >>>>> > https://github.com/theforeman/tool_belt. >>>>> > >>>>> > My proposal is to go forward with 2. Give Jenkins permissions to do >>>>> all >>>>> > of the actions needed, and whoever is the release nanny, ideally only >>>>> > has to make sure all of the steps are moving forward. If something >>>>> > breaks, figure out how to fix it for the next release. >>>>> > >>>>> > This would mean making a few extra jobs before and after the current >>>>> > release pipeline. In my opinion, it's the way to go to ensure anyone >>>>> can >>>>> > take over this responsibility. >>>>> > >>>>> > At this moment, we are in a situation where only people who >>>>> > mostly have permissions everywhere can successfully do a release >>>>> without >>>>> > asking many people for favors. >>>>> > >>>>> > Personally if we complete this, I see it as a big win as it would >>>>> dwarf >>>>> > our bus factor for release managers & allow us to release at any pace >>>>> we >>>>> > desire (right now it's slow because we can't truly release things from >>>>> > one day to the next due to the work involved). >>>>> > >>>>> > Thoughts? >>>>> > >>>>> > Here's the list of permissions: >>>>> > >>>>> > ---------------- >>>>> > >>>>> > Github: >>>>> > - Push in foreman, foreman-selinux, foreman-installer, >>>>> > smart-proxy, foreman-infra, foreman-packaging >>>>> > >>>>> > Transifex - >>>>> > - Allow to change the auto-update URL to point to latest -stable >>>>> > branch >>>>> > >>>>> > Redmine - >>>>> > - Create new "Found in Release" version >>>>> > >>>>> > Jenkins - >>>>> > - Modify jobs >>>>> > - Run jobs >>>>> > >>>>> > Koji - >>>>> > - Create tags >>>>> > - SSH access to update the mash scripts >>>>> > - Create packages >>>>> > - Tag builds >>>>> > >>>>> > Repository servers >>>>> > - ssh in deb.theforeman.org >>>>> > - ssh in yum.theforeman.org >>>>> > >>>>> > Announcements - >>>>> > - Post to foreman-announce >>>>> > - Merge access in theforeman.org >>>>> > - Change IRC message >>>>> > - Publish in Twitter, G+ >>>>> > >>>>> > --------------- >>>>> > >>>>> > -- >>>>> > Daniel Lobato Garcia >>>>> > >>>>> > @dLobatog >>>>> > blog.daniellobato.me >>>>> > daniellobato.me >>>>> > >>>>> > GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D >>>>> 6DE30 >>>>> > Keybase: https://keybase.io/elobato >>>>> > >>>>> > -- >>>>> > You received this message because you are subscribed to the Google >>>>> Groups "foreman-dev" group. >>>>> > To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to foreman-dev...@googlegroups.com . >>>>> > For more options, visit https://groups.google.com/d/optout. >>>>> >>>>> >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "foreman-dev" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to foreman-dev+unsubscribe@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "foreman-dev" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to foreman-dev+unsubscribe@googlegroups.com. >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > > >-- >Eric D. Helms >Red Hat Engineering > >-- >You received this message because you are subscribed to the Google Groups "foreman-dev" group. >To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com. >For more options, visit https://groups.google.com/d/optout.