Hosting the foreman CI jos at ci.centos.org

Development
1

hi,

We would like to offer resources at ci.centos.org to host the Foreman CI.

http://ci.centos.org/

there isnt a lot visible there at the moment, but ~ 12 projects are
onramping into that properly. Libvirt folks just went live on the 2nd
April. RDO will be live mid next week.

We have a consolidated facility of 128 physical nodes, that are
provisioned and managed as cattle - the test provisioners are encourated
to request and then abuse baremetal. There are intel and amd cpu's, 4 gb
of ram per code, backed by ssd storage, and upto 6 network devices per
machine.

Couple of links to get folks started up :
http://wiki.centos.org/QaWiki/CI has an overview that should give you a
fair idea on how things are setup and
http://wiki.centos.org/QaWiki/PubHardware has details on the hardware
resources backing the facility.

http://wiki.centos.org/QaWiki/CI/Duffy has details on the ultra simple
'api' for requesting and consuming hardware. We use this to maintain a
cache of 'freshly installed' machines, so requests can be completed in
real time. Also, the backing hardware platform does not have any form of
ACL in their management layers, Duffy helps bridge that gap for us.

How I'd imagine foreman would setup is that we would setup a persistant
VM, that gets deployed as the jenkins-slave, we get the slave setup in
the ci.centos.org jenkins interface, and you guys ( the foreman ci folks
) can then use the Duffy interface to request and run tests as needed.
We only baremetal provide CentOS5/6/7 - but dont mind hosting other
distro's in VM's.

Our ask is that artifacts from the build streams be made available via
something ( we dont care what, we can iteratively get better at this -
maybe rpms repos would be a good starting point ) - so that other
projects wanting to daisy chain as downstream deps of the content are
able to do so easily. We can provide storage outside of the
ci.centos.org facility for these artifacts.

We are ready to go pretty much right now, and can start getting content
online within a week or so. Next steps would be to work through a
technical layout of what the requirements are and how best we are able
to meet those. We would also ask for the project to nominate a few
people as liason for us to interface with.

Happy to have the conversation on the foreman-dev list.

Regards,

··· -- Karanbir Singh +44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh GnuPG Key : http://www.karan.org/publickey.asc
2

> hi,
>
> We would like to offer resources at ci.centos.org to host the Foreman CI.
>
> http://ci.centos.org/
>
> there isnt a lot visible there at the moment, but ~ 12 projects are
> onramping into that properly. Libvirt folks just went live on the 2nd
> April. RDO will be live mid next week.
>
> We have a consolidated facility of 128 physical nodes, that are
> provisioned and managed as cattle - the test provisioners are encourated
> to request and then abuse baremetal. There are intel and amd cpu's, 4 gb
> of ram per code, backed by ssd storage, and upto 6 network devices per
> machine.
>
> Couple of links to get folks started up :
> http://wiki.centos.org/QaWiki/CI has an overview that should give you a
> fair idea on how things are setup and
> http://wiki.centos.org/QaWiki/PubHardware has details on the hardware
> resources backing the facility.
>
> http://wiki.centos.org/QaWiki/CI/Duffy has details on the ultra simple
> 'api' for requesting and consuming hardware. We use this to maintain a
> cache of 'freshly installed' machines, so requests can be completed in
> real time. Also, the backing hardware platform does not have any form of
> ACL in their management layers, Duffy helps bridge that gap for us.

This all sounds potentially useful. Today some of the tests we do are
against "nightly" builds and releases and run on Rackspace Cloud via
Vagrant, but this could be changed.

The main benefit for us might be that it would reduce the amount of our
Rackspace allowance used for test VMs which could be used instead for
our dedicated slaves to run unit tests.

> How I'd imagine foreman would setup is that we would setup a persistant
> VM, that gets deployed as the jenkins-slave, we get the slave setup in
> the ci.centos.org jenkins interface, and you guys ( the foreman ci folks
> ) can then use the Duffy interface to request and run tests as needed.
> We only baremetal provide CentOS5/6/7 - but dont mind hosting other
> distro's in VM's.

We run against CentOS, Debian and Ubuntu so would need the multi-distro
support from VMs if we were thinking of moving our release tests. Would
the support for running VMs be provided via Duffy, or we'd spin up a
bare metal machine and run VMs on top of that?

Jenkins has support (or a plugin) for running a job on a remote Jenkins
server, so we could use this to delegate jobs from our own over to the
CentOS CI server, which would make it easier to integrate to our current
release pipelines etc.

Moving our unit tests could be problematic unless we changed how they
run, since we require quite a bit of software (Ruby stacks) and some
services (database servers) to execute them. My only thought here is if
we spun up hosts with Docker containers or some other prebuilt images
for PR testing, but it probably isn't worth the complexity.

> Our ask is that artifacts from the build streams be made available via
> something ( we dont care what, we can iteratively get better at this -
> maybe rpms repos would be a good starting point ) - so that other
> projects wanting to daisy chain as downstream deps of the content are
> able to do so easily. We can provide storage outside of the
> ci.centos.org facility for these artifacts.

We publish these already today at http://yum.theforeman.org/nightly/ and
on deb.theforeman.org, and would probably continue to do it this way.
These are pushed at the end of a successful nightly test cycle.

Cheers,

··· On 04/04/15 18:33, Karanbir Singh wrote:

–
Dominic Cleal
Red Hat Engineering

3

Hi,

> This all sounds potentially useful. Today some of the tests we do are
> against "nightly" builds and releases and run on Rackspace Cloud via
> Vagrant, but this could be changed.
>
> The main benefit for us might be that it would reduce the amount of our
> Rackspace allowance used for test VMs which could be used instead for
> our dedicated slaves to run unit tests.

So in this model : you would use the ci.centos.org jenkins master, and
setup slaves in the rackspace infra to run unit tests, per commit - and
scondly setup slaves inside ci.centos.org infra to run nightly, wider
scoped tests. Is that right ? We might need to workout how the security
process will work with slaves outside of the infra.

Can I then request you to start by filing a request here :
http://bugs.centos.org/bug_report_page.php against project: Buildsystem
and Category Community Buildsys

> We run against CentOS, Debian and Ubuntu so would need the multi-distro
> support from VMs if we were thinking of moving our release tests. Would
> the support for running VMs be provided via Duffy, or we'd spin up a
> bare metal machine and run VMs on top of that?

At the moment, we can only do baremetal, but we treat them like cattle!
There is an ongoing conversation around getting a cloud controller setup
inside the infra, allowing for arbitary images, with arbitary(ish)
instance sizes to be brought up on demand - but we are a month or more
away from that being available.

> Jenkins has support (or a plugin) for running a job on a remote Jenkins
> server, so we could use this to delegate jobs from our own over to the
> CentOS CI server, which would make it easier to integrate to our current
> release pipelines etc.
>
> Moving our unit tests could be problematic unless we changed how they
> run, since we require quite a bit of software (Ruby stacks) and some
> services (database servers) to execute them. My only thought here is if
> we spun up hosts with Docker containers or some other prebuilt images
> for PR testing, but it probably isn't worth the complexity.

okay, lets start with the easy bits.

> We publish these already today at http://yum.theforeman.org/nightly/ and
> on deb.theforeman.org, and would probably continue to do it this way.
> These are pushed at the end of a successful nightly test cycle.

sounds good!

··· On 04/09/2015 02:22 PM, Dominic Cleal wrote:

–
Karanbir Singh
+44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
GnuPG Key : http://www.karan.org/publickey.asc

4

> Hi,
>
>> This all sounds potentially useful. Today some of the tests we do are
>> against "nightly" builds and releases and run on Rackspace Cloud via
>> Vagrant, but this could be changed.
>>
>> The main benefit for us might be that it would reduce the amount of our
>> Rackspace allowance used for test VMs which could be used instead for
>> our dedicated slaves to run unit tests.
>
> So in this model : you would use the ci.centos.org jenkins master, and
> setup slaves in the rackspace infra to run unit tests, per commit - and
> scondly setup slaves inside ci.centos.org infra to run nightly, wider
> scoped tests. Is that right ? We might need to workout how the security
> process will work with slaves outside of the infra.

We would still use our own master for now, as we have a huge number of
jobs that we can't move simply.

We can run with both masters and have ci.theforeman.org trigger jobs on
ci.centos.org when needed.

> Can I then request you to start by filing a request here :
> http://bugs.centos.org/bug_report_page.php against project: Buildsystem
> and Category Community Buildsys

Will do shortly, once this release is out of the way.

>> We run against CentOS, Debian and Ubuntu so would need the multi-distro
>> support from VMs if we were thinking of moving our release tests. Would
>> the support for running VMs be provided via Duffy, or we'd spin up a
>> bare metal machine and run VMs on top of that?
>
> At the moment, we can only do baremetal, but we treat them like cattle!
> There is an ongoing conversation around getting a cloud controller setup
> inside the infra, allowing for arbitary images, with arbitary(ish)
> instance sizes to be brought up on demand - but we are a month or more
> away from that being available.

Yes, and as discussed on #centos-devel, we can perhaps do the bare metal
and VM aspect if needed.

··· On 14/04/15 11:29, Karanbir Singh wrote: > On 04/09/2015 02:22 PM, Dominic Cleal wrote:

–
Dominic Cleal
Red Hat Engineering

5

>
> We would still use our own master for now, as we have a huge number of
> jobs that we can't move simply.
>
> We can run with both masters and have ci.theforeman.org trigger jobs on
> ci.centos.org when needed.
>

lets start with that

>> Can I then request you to start by filing a request here :
>> http://bugs.centos.org/bug_report_page.php against project: Buildsystem
>> and Category Community Buildsys
>
> Will do shortly, once this release is out of the way.
>
…
>
> Yes, and as discussed on #centos-devel, we can perhaps do the bare metal
> and VM aspect if needed.
>

sounds good, I am going to leave this with you - we have some machines
nominated for the foreman jobs, and I will leave those as is as well.
Let me know once you have the issue request open and we can move on that.

··· On 20/04/15 08:41, Dominic Cleal wrote:

–
Karanbir Singh
+44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
GnuPG Key : http://www.karan.org/publickey.asc

6

I opened http://bugs.centos.org/view.php?id=8864 just now.

··· On 27/04/15 17:16, Karanbir Singh wrote: > On 20/04/15 08:41, Dominic Cleal wrote: >> >> We would still use our own master for now, as we have a huge number of >> jobs that we can't move simply. >> >> We can run with both masters and have ci.theforeman.org trigger jobs on >> ci.centos.org when needed. >> > > lets start with that > > >>> Can I then request you to start by filing a request here : >>> http://bugs.centos.org/bug_report_page.php against project: Buildsystem >>> and Category Community Buildsys >> >> Will do shortly, once this release is out of the way. >> > ... >> >> Yes, and as discussed on #centos-devel, we can perhaps do the bare metal >> and VM aspect if needed. >> > > sounds good, I am going to leave this with you - we have some machines > nominated for the foreman jobs, and I will leave those as is as well. > Let me know once you have the issue request open and we can move on that.

–
Dominic Cleal
Red Hat Engineering

7

ticket above is already resolved, what else is still open?

thanks,
Ohad

··· On Mon, Jun 8, 2015 at 3:23 PM, Dominic Cleal wrote:

On 27/04/15 17:16, Karanbir Singh wrote:

On 20/04/15 08:41, Dominic Cleal wrote:

We would still use our own master for now, as we have a huge number of
jobs that we can’t move simply.

We can run with both masters and have ci.theforeman.org trigger jobs on
ci.centos.org when needed.

lets start with that

Can I then request you to start by filing a request here :
http://bugs.centos.org/bug_report_page.php against project:
Buildsystem
and Category Community Buildsys

Will do shortly, once this release is out of the way.

…

Yes, and as discussed on #centos-devel, we can perhaps do the bare metal
and VM aspect if needed.

sounds good, I am going to leave this with you - we have some machines
nominated for the foreman jobs, and I will leave those as is as well.
Let me know once you have the issue request open and we can move on that.

I opened http://bugs.centos.org/view.php?id=8864 just now.

Does anyone knows what is the status of using CentOS infrastructure? the

8

I believe its all setup from our end, and there is a test job where
Dominic has setup the mechanics as well.

let me know if I can help with anything else,

··· On 19/10/15 14:04, Ohad Levy wrote: > I opened http://bugs.centos.org/view.php?id=8864 just now. > > > Does anyone knows what is the status of using CentOS infrastructure? the > ticket above is already resolved, what else is still open?

–
Karanbir Singh
+44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
GnuPG Key : http://www.karan.org/publickey.asc

9

Writing jobs and updating jobs to use the infrastructure.
http://projects.theforeman.org/projects/foreman/wiki/Jenkins#CentOS-QA-service
discusses some of it.

I have a foreman-infra branch in progress which duplicates our
systest_foreman type job onto ci.centos.org infra, though it's
implemented quite differently with libvirt. It's rather complex though,
which makes me hesitant to propose it to be permanent.

··· On 19/10/15 14:04, Ohad Levy wrote: > Does anyone knows what is the status of using CentOS infrastructure? the > ticket above is already resolved, what else is still open?

–
Dominic Cleal
dominic@cleal.org

10

There's a plugin which I was waiting for at one point, it doesn't look
like it's installed at the moment. See
https://lists.centos.org/pipermail/ci-users/2015-August/000002.html

··· On 22/10/15 18:14, Karanbir Singh wrote: > On 19/10/15 14:04, Ohad Levy wrote: >> I opened http://bugs.centos.org/view.php?id=8864 just now. >> >> >> Does anyone knows what is the status of using CentOS infrastructure? the >> ticket above is already resolved, what else is still open? > > I believe its all setup from our end, and there is a test job where > Dominic has setup the mechanics as well. > > let me know if I can help with anything else,

–
Dominic Cleal
dominic@cleal.org

11

> > Does anyone knows what is the status of using CentOS infrastructure? the
> > ticket above is already resolved, what else is still open?
>
> Writing jobs and updating jobs to use the infrastructure.
>
> Jenkins - Foreman
> discusses some of it.
>
> I have a foreman-infra branch in progress which duplicates our
> systest_foreman type job onto ci.centos.org infra, though it's
> implemented quite differently with libvirt. It's rather complex though,
> which makes me hesitant to propose it to be permanent.
>

I'd like to take and build on what you've done here to off load the Katello
systests and look into adding more complex full plugin composition tests to
this infrastructure. I have familiarized myself with the general process
and gotten an account and access setup. I had a few questions to get me up
to speed with where you had taken this and what I need to build out:

  • When you were testing this, were you uploading the job definitions
    manually or do you have this built into the normal job upload workflow?
    • If manual, do you have an idea of how to manage the job definitions
      across the two CI systems already?
  • Do you mind sharing your branch for a starting place?
  • Do you forsee any resource bottle neck with using the remote trigger
    plugin and keeping a job running to check the status of the job in
    ci.centos.org? I was thinking of ways to re-work the pipeline so that the
    systests are not in the middle but rather the starting point but if this is
    not an issue i will abandon that thought.

Eric

··· On Fri, Oct 23, 2015 at 3:16 AM, Dominic Cleal wrote: > On 19/10/15 14:04, Ohad Levy wrote:

–
Dominic Cleal
dominic@cleal.org

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University

12

let me look into this, and get back

··· On 23/10/15 08:14, Dominic Cleal wrote: > On 22/10/15 18:14, Karanbir Singh wrote: >> On 19/10/15 14:04, Ohad Levy wrote: >>> I opened http://bugs.centos.org/view.php?id=8864 just now. >>> >>> >>> Does anyone knows what is the status of using CentOS infrastructure? the >>> ticket above is already resolved, what else is still open? >> >> I believe its all setup from our end, and there is a test job where >> Dominic has setup the mechanics as well. >> >> let me know if I can help with anything else, > > There's a plugin which I was waiting for at one point, it doesn't look > like it's installed at the moment. See > https://lists.centos.org/pipermail/ci-users/2015-August/000002.html >

–
Karanbir Singh
+44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
GnuPG Key : http://www.karan.org/publickey.asc

13

>
>
>
> > Does anyone knows what is the status of using CentOS infrastructure? the
> > ticket above is already resolved, what else is still open?
>
> Writing jobs and updating jobs to use the infrastructure.
> Jenkins - Foreman
> discusses some of it.
>
> I have a foreman-infra branch in progress which duplicates our
> systest_foreman type job onto ci.centos.org <http://ci.centos.org>
> infra, though it's
> implemented quite differently with libvirt. It's rather complex though,
> which makes me hesitant to propose it to be permanent.
>
>
> I'd like to take and build on what you've done here to off load the
> Katello systests and look into adding more complex full plugin
> composition tests to this infrastructure. I have familiarized myself
> with the general process and gotten an account and access setup. I had a
> few questions to get me up to speed with where you had taken this and
> what I need to build out:
>
> * When you were testing this, were you uploading the job definitions
> manually or do you have this built into the normal job upload workflow?
> - If manual, do you have an idea of how to manage the job definitions
> across the two CI systems already?

It was using JJB to manage job definitions, in the same way as we do
with ci.theforeman.org, though I was running it locally while testing.

> * Do you mind sharing your branch for a starting place?

https://github.com/domcleal/foreman-infra/commits/jjb-centos is my
branch for the changes.

> * Do you forsee any resource bottle neck with using the remote trigger
> plugin and keeping a job running to check the status of the job in
> ci.centos.org <http://ci.centos.org>? I was thinking of ways to re-work
> the pipeline so that the systests are not in the middle but rather the
> starting point but if this is not an issue i will abandon that thought.

I don't know, isn't that going to depend on the number and when they run?

··· On 11/04/16 19:51, Eric D Helms wrote: > On Fri, Oct 23, 2015 at 3:16 AM, Dominic Cleal > wrote: > On 19/10/15 14:04, Ohad Levy wrote:

–
Dominic Cleal
dominic@cleal.org

14

I believe this has now been resolved, ref:
https://lists.centos.org/pipermail/ci-users/2015-October/000040.html

let me know if you hit any further blockers, of if we can help in anyway
get things rolling.

regards,

··· On 23/10/15 23:38, Karanbir Singh wrote: > On 23/10/15 08:14, Dominic Cleal wrote: >> On 22/10/15 18:14, Karanbir Singh wrote: >>> On 19/10/15 14:04, Ohad Levy wrote: >>>> I opened http://bugs.centos.org/view.php?id=8864 just now. >>>> >>>> >>>> Does anyone knows what is the status of using CentOS infrastructure? the >>>> ticket above is already resolved, what else is still open? >>> >>> I believe its all setup from our end, and there is a test job where >>> Dominic has setup the mechanics as well. >>> >>> let me know if I can help with anything else, >> >> There's a plugin which I was waiting for at one point, it doesn't look >> like it's installed at the moment. See >> https://lists.centos.org/pipermail/ci-users/2015-August/000002.html

–
Karanbir Singh
+44-207-0999389 | http://www.karan.org/ | twitter.com/kbsingh
GnuPG Key : http://www.karan.org/publickey.asc

15

Could we add the

to our Jenkins? I've created a base job to offload the Katello systest but
would like to run it in parallel a bit before swapping out entirely.

Eric

··· On Tue, Apr 12, 2016 at 3:02 AM, Dominic Cleal wrote:

On 11/04/16 19:51, Eric D Helms wrote:

On Fri, Oct 23, 2015 at 3:16 AM, Dominic Cleal <dominic@cleal.org > > mailto:dominic@cleal.org> wrote:

On 19/10/15 14:04, Ohad Levy wrote:
> Does anyone knows what is the status of using CentOS

infrastructure? the

> ticket above is already resolved, what else is still open?

Writing jobs and updating jobs to use the infrastructure.

Jenkins - Foreman

discusses some of it.

I have a foreman-infra branch in progress which duplicates our
systest_foreman type job onto ci.centos.org <http://ci.centos.org>
infra, though it's
implemented quite differently with libvirt.  It's rather complex

though,

which makes me hesitant to propose it to be permanent.

I’d like to take and build on what you’ve done here to off load the
Katello systests and look into adding more complex full plugin
composition tests to this infrastructure. I have familiarized myself
with the general process and gotten an account and access setup. I had a
few questions to get me up to speed with where you had taken this and
what I need to build out:

  • When you were testing this, were you uploading the job definitions
    manually or do you have this built into the normal job upload workflow?
    • If manual, do you have an idea of how to manage the job definitions
      across the two CI systems already?

It was using JJB to manage job definitions, in the same way as we do
with ci.theforeman.org, though I was running it locally while testing.

  • Do you mind sharing your branch for a starting place?

Commits · domcleal/foreman-infra · GitHub is my
branch for the changes.

  • Do you forsee any resource bottle neck with using the remote trigger
    plugin and keeping a job running to check the status of the job in
    ci.centos.org http://ci.centos.org? I was thinking of ways to re-work
    the pipeline so that the systests are not in the middle but rather the
    starting point but if this is not an issue i will abandon that thought.

I don’t know, isn’t that going to depend on the number and when they run?

–
Dominic Cleal
dominic@cleal.org

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University

16

Done.

··· On 12/05/16 16:12, Eric D Helms wrote: > Could we add > the https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Remote+Trigger+Plugin > to our Jenkins? I've created a base job to offload the Katello systest > but would like to run it in parallel a bit before swapping out entirely.

–
Dominic Cleal
dominic@cleal.org

17

Thanks for adding the plugin. Could you also configure a remote host
pointing to ci.centos.org within the plugin's configuration section?

··· On Sat, May 14, 2016 at 4:35 AM, Dominic Cleal wrote:

On 12/05/16 16:12, Eric D Helms wrote:

Could we add
the
Parameterized Remote Trigger
to our Jenkins? I’ve created a base job to offload the Katello systest
but would like to run it in parallel a bit before swapping out entirely.

Done.

–
Dominic Cleal
dominic@cleal.org

–
You received this message because you are subscribed to the Google Groups
“foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

–
Eric D. Helms
Red Hat Engineering
Ph.D. Student - North Carolina State University

18

Sorry, I missed that. Added now.

··· On 18/05/16 00:26, Eric D Helms wrote: > Thanks for adding the plugin. Could you also configure a remote host > pointing to ci.centos.org within the plugin's > configuration section?

–
Dominic Cleal
dominic@cleal.org