(This is mainly aimed at Sam, but feel free to contribute…)
Ok, so in the last two weeks of frantic package building, I've
discovered two things:
- The repo is a mess
- The repo is not scalable
The was demonstrated to me perfectly on Friday, trying to fix the
Ubuntu issues. I had 4 separate VMs to test on - a 2x2 matrix of
Debian vs Ubuntu and 32bit vs 64bit. Took far too long. So, we need
something better, and it needs to be comprised of three things:
- Source storage
···
-------------------------
Right now the Foreman Debian build files are in Git, and that’s great.
We need to extend this. Thinking ahead to contributing some of the
gems upstream to Debian, and thus looking at
http://wiki.debian.org/Teams/Ruby/Packaging we should also store the
gem builds in git. Seems easy enough to create
github.com/theforeman/debian-packaging.
We have 3 rough classes of package:
Foreman/Proxy: We need to re-write the packaging with git-buildpackage
(http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html)
Ruby Gems: As above link, these are fairly easy, especially if we
start by disabling the tests 
Dependencies: Notably (for Squeeze) a backport of bundler,
rubygems-integration, and gem2deb-0.3.
- Building packages
We need to be building all these packages automatically, according to
some ruleset. The natural choice of buildsystem for Debian-like
systems is PBuilder
(http://www.netfort.gr.jp/~dancer/software/pbuilder-doc/pbuilder-doc.html)
which I have used myself several times. Pbuilder creates a clean
tarball of each OS we want to build for, and can create 32bit packages
as well, so with 4 tarballs, we can replicate the setup I had on
Friday. These clean tarballs will have a single modification - they
will have the foreman deb repo added to their sources. I view the
build flow being something like:
-
Have a “master package list” stored somewhere (Git). The packages
will be built from this list list, in order. Thus dependencies will be
top of the file.
-
Pbuilder is used to build the package using the git sources (git
clone … && pbuilder --base -.tgz .dsc)
-
The complete package is uploaded to the Foreman repo to be
available for the next package to be built
-
Build the next package (return to Step 1 until all are built)
-
Testing (this is the bit I’m aiming at you Sam ;P)
Once all the packages are built, we need to trigger some kind of test
system. I guess Jenkins is good here? I honestly am not sure how best
to test the packages. Ideas accepted
Optional Stuff
Right now only the nightlies are fully automated - uploads to testing
or stable are manual. My aim is to implement the above for the
nightlies over the next week or two, and then look at how we can
re-use the same infrastructure for auto-promoting to testing/stable.
The main difference is that we actually care about the package version
numbers in stable/testing so extra care has to be taken to handle that
properly.
The repo format is also up for discussion. Right now we use a single
"main" repo, but now we find ourselves needing to support multiple
OSes. I have no immediate suggestions beyond simply using two URLs
(deb.theforman.org/{debian,ubuntu}) but I’ll do a little research.
Quite a lot to do, but perhaps not as much as it could be. Thoughts?
Greg
> (This is mainly aimed at Sam, but feel free to contribute…)
>
> Ok, so in the last two weeks of frantic package building, I've
> discovered two things:
>
> 1) The repo is a mess
> 2) The repo is not scalable
>
> The was demonstrated to me perfectly on Friday, trying to fix the
> Ubuntu issues. I had 4 separate VMs to test on - a 2x2 matrix of
> Debian vs Ubuntu and 32bit vs 64bit. Took far too long. So, we need
> something better, and it needs to be comprised of three things:
>
> 1) Source storage
> -------------------------
>
> Right now the Foreman Debian build files are in Git, and that's great.
> We need to extend this. Thinking ahead to contributing some of the
> gems upstream to Debian, and thus looking at
> http://wiki.debian.org/Teams/Ruby/Packaging we should also store the
> gem builds in git. Seems easy enough to create
> github.com/theforeman/debian-packaging.
>
> We have 3 rough classes of package:
>
> Foreman/Proxy: We need to re-write the packaging with git-buildpackage
> (http://honk.sigxcpu.org/projects/git-buildpackage/manual-html/gbp.html)
> Ruby Gems: As above link, these are fairly easy, especially if we
> start by disabling the tests
> Dependencies: Notably (for Squeeze) a backport of bundler,
> rubygems-integration, and gem2deb-0.3.
>
> 2) Building packages
> ------------------------------
>
> We need to be building all these packages automatically, according to
> some ruleset. The natural choice of buildsystem for Debian-like
> systems is PBuilder
> (http://www.netfort.gr.jp/~dancer/software/pbuilder-doc/pbuilder-doc.html)
> which I have used myself several times. Pbuilder creates a clean
> tarball of each OS we want to build for, and can create 32bit packages
> as well, so with 4 tarballs, we can replicate the setup I had on
> Friday. These clean tarballs will have a single modification - they
> will have the foreman deb repo added to their sources. I view the
> build flow being something like:
>
> 1) Have a "master package list" stored somewhere (Git). The packages
> will be built from this list list, in order. Thus dependencies will be
> top of the file.
> 2) Pbuilder is used to build the package using the git sources (git
> clone … && pbuilder --base <os>-<arch>.tgz <package-name>.dsc)
> 3) The complete package is uploaded to the Foreman repo to be
> available for the next package to be built
> 4) Build the next package (return to Step 1 until all are built)
>
> 3) Testing (this is the bit I'm aiming at you Sam ;P)
> --------------
>
> Once all the packages are built, we need to trigger some kind of test
> system. I guess Jenkins is good here? I honestly am not sure how best
> to test the packages. Ideas accepted
>
> Optional Stuff
> --------------------
>
> Right now only the nightlies are fully automated - uploads to testing
> or stable are manual. My aim is to implement the above for the
> nightlies over the next week or two, and then look at how we can
> re-use the same infrastructure for auto-promoting to testing/stable.
> The main difference is that we actually care about the package version
> numbers in stable/testing so extra care has to be taken to handle that
> properly.
>
> The repo format is also up for discussion. Right now we use a single
> "main" repo, but now we find ourselves needing to support multiple
> OSes. I have no immediate suggestions beyond simply using two URLs
> (deb.theforman.org/{debian,ubuntu}) but I'll do a little research.
>
> Quite a lot to do, but perhaps not as much as it could be. Thoughts?
>
does deb has some equiv of Koji? where people need to vote up for a package
before it gets promoted?
e.g. the ubuntu xx package wont be pushed into testing/stable unless xx
people gave it +1 ?
Ohad
···
On Mon, Aug 6, 2012 at 5:41 PM, Greg Sutcliffe wrote:
Greg
I'm not aware of any such thing. Time to port Pulp to Debian? 
Greg
···
On 7 August 2012 07:39, Ohad Levy wrote:
> does deb has some equiv of Koji? where people need to vote up for a package
> before it gets promoted?
>
> e.g. the ubuntu xx package wont be pushed into testing/stable unless xx
> people gave it +1 ?
I did some digging on this yesterday and couldn't find anything like
Koji for Debian.
-Sam
···
On 08/08/2012 08:28 AM, Greg Sutcliffe wrote:
> On 7 August 2012 07:39, Ohad Levy wrote:
>> does deb has some equiv of Koji? where people need to vote up for a package
>> before it gets promoted?
>>
>> e.g. the ubuntu xx package wont be pushed into testing/stable unless xx
>> people gave it +1 ?
> I'm not aware of any such thing. Time to port Pulp to Debian? :)
>
> Greg
We could create something like it if we really wanted to. Store all
the debian packaging in Git, with the packages auto-building for a
certain branch, and then use Git tools (such as Gerrit) to
up-vote/auto-merge to the build branch.
Probably more effort than it's worth though
Greg
Greg Sutcliffe
Operations Engineer
iBAHN Stirling
Logie Court
Stirling University Innovation Park
Stirling
Scotland, UK
FK9 4NF
T: +44 (0) 1786 455150
OpenPGP -> KeyID: CA0AEB93
···
On 08/08/12 13:34, Sam Kottler wrote:
> I did some digging on this yesterday and couldn't find anything
> like Koji for Debian.
>
> -Sam
They have buildd
https://buildd.debian.org/
http://www.debian.org/devel/buildd/
but is is private (for debian developers) only.
In fact you have to build in on you local machine before submitting. And
only if it is accepted, then it is build in buildd.
The drawback is that becoming Debian Developer is not easy task:
http://www.debian.org/devel/join/
I have already few packages in Debian, but I'm still not Debian
Developer. In fact I know collegues of mine, who maintain package (via
Sponsor) for years and he still does not get status of Debian Developer.
Package is propagated to testing usually within two weeks, but every
problem is stopper:
http://www.debian.org/doc/manuals/debian-faq/ch-ftparchives#s-testing
Hmm now I realize I'm speaking about Debian, while you ask about Ubuntu.
Mirek
···
On 8.8.2012 14:28, Greg Sutcliffe wrote:
> does deb has some equiv of Koji? where people need to vote up for a package
>> before it gets promoted?
>>
>> e.g. the ubuntu xx package wont be pushed into testing/stable unless xx
>> people gave it +1 ?
