Using Fedora Copr for RPM builds

I spoke to Miroslav Suchy today, who's one of the developers of the
Fedora Copr project (https://copr.fedoraproject.org/) about whether it's
suitable to replace Koji as our RPM build system.

I think it's quite promising as a replacement, as it's stabilised and
gathered a number of new features over the past few months. Balanced
against the risk and work involved in running a Koji instance ourselves,
I'd say it's an improvement.

The main problem seems to be that it's missing a way for us to branch
for Foreman releases. Today we create and clone new Koji tags and
continue to build simultaneously into the nightly tag from rpm/develop
and the stable tags from rpm/1.* branches in foreman-packaging.

As it stands I think we'd need to create a new Copr project for a stable
release and mass-build into it to bootstrap a new major release.
Miroslav's going to look into the possibility of adding a copy function
to Copr, which could let us copy a nightly Copr project to a stable
project in a similar way.

We'd also need to make changes to our tito configuration, package test
and publish scripts (mostly in Jenkins), but this shouldn't be too
difficult, since it will generate repositories in a reasonably similar
way, which can be signed and copied to yum.theforeman.org.

Unless there are any other issues that come to mind, I'll probably
revisit this if/when a copy feature is available and see about migrating
our RPM builds over.

··· -- Dominic Cleal dominic@cleal.org

Any insight into the performance of the Copr service? Will we run into
bottlenecks when submitting lots of builds or delays in processing?

Eric

··· On Wed, Oct 14, 2015 at 3:06 PM, Dominic Cleal wrote:

I spoke to Miroslav Suchy today, who’s one of the developers of the
Fedora Copr project (https://copr.fedoraproject.org/) about whether it’s
suitable to replace Koji as our RPM build system.

I think it’s quite promising as a replacement, as it’s stabilised and
gathered a number of new features over the past few months. Balanced
against the risk and work involved in running a Koji instance ourselves,
I’d say it’s an improvement.

The main problem seems to be that it’s missing a way for us to branch
for Foreman releases. Today we create and clone new Koji tags and
continue to build simultaneously into the nightly tag from rpm/develop
and the stable tags from rpm/1.* branches in foreman-packaging.

As it stands I think we’d need to create a new Copr project for a stable
release and mass-build into it to bootstrap a new major release.
Miroslav’s going to look into the possibility of adding a copy function
to Copr, which could let us copy a nightly Copr project to a stable
project in a similar way.

We’d also need to make changes to our tito configuration, package test
and publish scripts (mostly in Jenkins), but this shouldn’t be too
difficult, since it will generate repositories in a reasonably similar
way, which can be signed and copied to yum.theforeman.org.

Unless there are any other issues that come to mind, I’ll probably
revisit this if/when a copy feature is available and see about migrating
our RPM builds over.


Dominic Cleal
dominic@cleal.org


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University

That's possible, but I think as it stands we'll have at least the same
number of builders available to us as we currently have permanently
running on Koji.

Mirek said that they have up to about 25 builders right now, and my
understanding is that it tries to apply fair share type scheduling
between users. I think he mentioned it can be tweaked if we find it's
not working well.

My experiences running a few nightly builds through the service in the
early days was that of long delays, but of late it's improved
significantly. The biggest risk in this area is that the service either
goes down or is swamped, affecting nightly builds or delaying a release.
My feeling is that if we lost our "snowflake" Koji instance, we'd be
even worse off.

The other interesting factor might be that we'll no longer have the huge
performance problem in Koji of regenerating inherited build repos after
every build. It'll only be regenerating the repo containing our
packages, instead of using huge mergerepo jobs which merge into the base
OS and other external repos (slow!).

··· On 14/10/15 16:37, Eric D Helms wrote: > Any insight into the performance of the Copr service? Will we run into > bottlenecks when submitting lots of builds or delays in processing?


Dominic Cleal
dominic@cleal.org

Great to hear!

Do you think it would be worth our time to implement and update our scripts
for nightly Copr builds to do some up front work and ensure that our
workflows work in practice? That we aren't missing any hidden dragons that
we'd need fixed before we could use it full time?

Eric

··· On Wed, Oct 14, 2015 at 4:43 PM, Dominic Cleal wrote:

On 14/10/15 16:37, Eric D Helms wrote:

Any insight into the performance of the Copr service? Will we run into
bottlenecks when submitting lots of builds or delays in processing?

That’s possible, but I think as it stands we’ll have at least the same
number of builders available to us as we currently have permanently
running on Koji.

Mirek said that they have up to about 25 builders right now, and my
understanding is that it tries to apply fair share type scheduling
between users. I think he mentioned it can be tweaked if we find it’s
not working well.

My experiences running a few nightly builds through the service in the
early days was that of long delays, but of late it’s improved
significantly. The biggest risk in this area is that the service either
goes down or is swamped, affecting nightly builds or delaying a release.
My feeling is that if we lost our “snowflake” Koji instance, we’d be
even worse off.

The other interesting factor might be that we’ll no longer have the huge
performance problem in Koji of regenerating inherited build repos after
every build. It’ll only be regenerating the repo containing our
packages, instead of using huge mergerepo jobs which merge into the base
OS and other external repos (slow!).


Dominic Cleal
dominic@cleal.org


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University

Yeah, that would be great. I was thinking to try a full rebuild soon,
as I've got the order for Foreman packages (since doing tfm).

··· On 14/10/15 17:07, Eric D Helms wrote: > Great to hear! > > Do you think it would be worth our time to implement and update our > scripts for nightly Copr builds to do some up front work and ensure that > our workflows work in practice? That we aren't missing any hidden > dragons that we'd need fixed before we could use it full time?


Dominic Cleal
dominic@cleal.org