Proposal: Move Foreman Jenkins Jobs to foreman-ci repository

All,

I brought this idea up in a separate thread, but want to formalize it into
it's own direct proposal. As of today, the Jenkins Job (JJB) configurations
live buried inside the foreman-infra repository. I believe this makes them
hard to discover [1] and awkward to work with being inside a puppet module.
My proposal is:

  1. Create foreman-ci github repository
  2. Move everything under [1] to foreman-ci
  3. Update the jenkins_job_builder puppet_module to clone this new
    repository

Further, I think this will allow CI focused work to happen and be separate
from the maintenance of our community infrastructure.

Eric

[1]
https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/jenkins_job_builder/files/theforeman.org

··· -- Eric D. Helms Red Hat Engineering

+1 from me, its very hard to discover today.

Greg

··· On 08/11/17 12:35, Eric D Helms wrote: > All, > > I brought this idea up in a separate thread, but want to formalize it into > it's own direct proposal. As of today, the Jenkins Job (JJB) configurations > live buried inside the foreman-infra repository. I believe this makes them > hard to discover [1] and awkward to work with being inside a puppet module. > My proposal is: > > 1) Create foreman-ci github repository > 2) Move everything under [1] to foreman-ci > 3) Update the jenkins_job_builder puppet_module to clone this new > repository > > Further, I think this will allow CI focused work to happen and be separate > from the maintenance of our community infrastructure.

> > All,
> >
> > I brought this idea up in a separate thread, but want to formalize it
> into
> > it's own direct proposal. As of today, the Jenkins Job (JJB)
> configurations
> > live buried inside the foreman-infra repository. I believe this makes
> them
> > hard to discover [1] and awkward to work with being inside a puppet
> module.
> > My proposal is:
> >
> > 1) Create foreman-ci github repository
> > 2) Move everything under [1] to foreman-ci
> > 3) Update the jenkins_job_builder puppet_module to clone this new
> > repository
> >
> > Further, I think this will allow CI focused work to happen and be
> separate
> > from the maintenance of our community infrastructure.
>
> +1 from me, its very hard to discover today.
>

+1 anything that makes it easier for a developer to understand the CI infra
better is a plus :slight_smile:

··· On Wed, Nov 8, 2017 at 3:42 PM, Greg Sutcliffe wrote: > On 08/11/17 12:35, Eric D Helms wrote:

Greg


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

We do something similar in upstream Pulp today with our jekins job templates, and it has worked out
really well for us. It sounds like a good idea.

··· On Wed, 2017-11-08 at 07:35 -0500, Eric D Helms wrote: > All, > > I brought this idea up in a separate thread, but want to formalize it into it's own direct > proposal. As of today, the Jenkins Job (JJB) configurations live buried inside the foreman-infra > repository. I believe this makes them hard to discover [1] and awkward to work with being inside a > puppet module. My proposal is: > > 1) Create foreman-ci github repository > 2) Move everything under [1] to foreman-ci > 3) Update the jenkins_job_builder puppet_module to clone this new repository > > Further, I think this will allow CI focused work to happen and be separate from the maintenance of > our community infrastructure. > > > Eric > > [1] https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/jenkins_job_builder/fil > es/theforeman.org >

>All,
>
>I brought this idea up in a separate thread, but want to formalize it into
>it's own direct proposal. As of today, the Jenkins Job (JJB) configurations
>live buried inside the foreman-infra repository. I believe this makes them
>hard to discover [1] and awkward to work with being inside a puppet module.
>My proposal is:
>
> 1) Create foreman-ci github repository
> 2) Move everything under [1] to foreman-ci
> 3) Update the jenkins_job_builder puppet_module to clone this new
>repository
>
>Further, I think this will allow CI focused work to happen and be separate
>from the maintenance of our community infrastructure.

+1 to making it more visible.

I'm not sure whether a separate repo or a top level directory in
foreman-infra is best. One benefit of a single repository is that
they're somewhat tightly linked: the JJB version and the templates we
use.

Would a we be able to move the directory to the top level and have a
symlink at the puppet module level? If that'd work I'd prefer that as a
first step since we wouldn't have to modify our current deployment
model.

··· On Wed, Nov 08, 2017 at 07:35:04AM -0500, Eric D Helms wrote:

Eric

[1]
https://github.com/theforeman/foreman-infra/tree/master/puppet/modules/jenkins_job_builder/files/theforeman.org

>
>> All,
>>
>> I brought this idea up in a separate thread, but want to formalize it into
>> it's own direct proposal. As of today, the Jenkins Job (JJB)
>> configurations
>> live buried inside the foreman-infra repository. I believe this makes them
>> hard to discover [1] and awkward to work with being inside a puppet
>> module.
>> My proposal is:
>>
>> 1) Create foreman-ci github repository
>> 2) Move everything under [1] to foreman-ci
>> 3) Update the jenkins_job_builder puppet_module to clone this new
>> repository
>>
>> Further, I think this will allow CI focused work to happen and be separate
>> from the maintenance of our community infrastructure.
>>
>
> +1 to making it more visible.
>
> I'm not sure whether a separate repo or a top level directory in
> foreman-infra is best. One benefit of a single repository is that they're
> somewhat tightly linked: the JJB version and the templates we use.
>

I don't know when you say template here. What template?

>
> Would a we be able to move the directory to the top level and have a
> symlink at the puppet module level? If that'd work I'd prefer that as a
> first step since we wouldn't have to modify our current deployment model.

If a separate repository, would it not be as simple as adding a git clone?

··· On Thu, Nov 9, 2017 at 9:00 AM, Ewoud Kohl van Wijngaarden < ewoud@kohlvanwijngaarden.nl> wrote: > On Wed, Nov 08, 2017 at 07:35:04AM -0500, Eric D Helms wrote:

Eric

[1]
https://github.com/theforeman/foreman-infra/tree/master/pupp
et/modules/jenkins_job_builder/files/theforeman.org


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering

>
>>
>>> All,
>>>
>>> I brought this idea up in a separate thread, but want to formalize it into
>>> it's own direct proposal. As of today, the Jenkins Job (JJB)
>>> configurations
>>> live buried inside the foreman-infra repository. I believe this makes them
>>> hard to discover 1 and awkward to work with being inside a puppet
>>> module.
>>> My proposal is:
>>>
>>> 1) Create foreman-ci github repository
>>> 2) Move everything under 1 to foreman-ci
>>> 3) Update the jenkins_job_builder puppet_module to clone this new
>>> repository
>>>
>>> Further, I think this will allow CI focused work to happen and be separate
>>> from the maintenance of our community infrastructure.
>>>
>>
>> +1 to making it more visible.
>>
>> I'm not sure whether a separate repo or a top level directory in
>> foreman-infra is best. One benefit of a single repository is that they're
>> somewhat tightly linked: the JJB version and the templates we use.
>>
>
>I don't know when you say template here. What template?

Sorry, that was a weird thing in my head: it's how we distribute the
jenkins jobs. Besides the plain text files we also have some slave
configs1 and they're templated. I don't know if those should be moved
too or translated into pure JJB files but it's part of how Jenkins is
configured.

>> Would a we be able to move the directory to the top level and have a
>> symlink at the puppet module level? If that'd work I'd prefer that as a
>> first step since we wouldn't have to modify our current deployment model.
>
>
>If a separate repository, would it not be as simple as adding a git clone?

Possibly, I don't have that much insight into the entire deployment
flow. Hence my preference of making small changes since our backlog is
big enough already. Of course if you're willing to take it on then be my
guest.

··· On Thu, Nov 09, 2017 at 09:04:55AM -0500, Eric D Helms wrote: >On Thu, Nov 9, 2017 at 9:00 AM, Ewoud Kohl van Wijngaarden < >ewoud@kohlvanwijngaarden.nl> wrote: >> On Wed, Nov 08, 2017 at 07:35:04AM -0500, Eric D Helms wrote:

Currently it's a Jenkins Job [1] that notices a change in the upstream
repo, and deploys it to the puppetmaster. Should be trivial enough to
make the changes you want there - adding a git submodule to the infra
repo might "just work", looking at it.

[1] http://ci.theforeman.org/job/deploy_puppet/

··· On 09/11/17 14:25, Ewoud Kohl van Wijngaarden wrote: >> If a separate repository, would it not be as simple as adding a git >> clone? > > Possibly, I don't have that much insight into the entire deployment > flow. Hence my preference of making small changes since our backlog is > big enough already. Of course if you're willing to take it on then be my > guest.

+1

It makes it better discoverable for new users/contributors and also
makes the repo easier to consume by others (SatelliteQE uses it).

··· On Wed, Nov 8, 2017 at 2:58 PM, Ohad Levy wrote: > > > On Wed, Nov 8, 2017 at 3:42 PM, Greg Sutcliffe > wrote: >> >> On 08/11/17 12:35, Eric D Helms wrote: >> > All, >> > >> > I brought this idea up in a separate thread, but want to formalize it >> > into >> > it's own direct proposal. As of today, the Jenkins Job (JJB) >> > configurations >> > live buried inside the foreman-infra repository. I believe this makes >> > them >> > hard to discover [1] and awkward to work with being inside a puppet >> > module. >> > My proposal is: >> > >> > 1) Create foreman-ci github repository >> > 2) Move everything under [1] to foreman-ci >> > 3) Update the jenkins_job_builder puppet_module to clone this new >> > repository >> > >> > Further, I think this will allow CI focused work to happen and be >> > separate >> > from the maintenance of our community infrastructure. >> >> +1 from me, its very hard to discover today. > > > +1 anything that makes it easier for a developer to understand the CI infra > better is a plus :) >> >> >> Greg >> >> -- >> You received this message because you are subscribed to the Google Groups >> "foreman-dev" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to foreman-dev+unsubscribe@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. > > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout.


Beste Grüße/Kind regards,

Evgeni Golov
Software Engineer


Red Hat GmbH, http://www.de.redhat.com/, Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O’Neill, Eric Shander

A git submodule might work - we wouldn't get automatic updates thus
doubling the work though. If you mean adding a separate repository to
the config then that could work well.

··· On Thu, Nov 09, 2017 at 02:37:10PM +0000, Greg Sutcliffe wrote: >On 09/11/17 14:25, Ewoud Kohl van Wijngaarden wrote: >>> If a separate repository, would it not be as simple as adding a git >>> clone? >> >> Possibly, I don't have that much insight into the entire deployment >> flow. Hence my preference of making small changes since our backlog is >> big enough already. Of course if you're willing to take it on then be my >> guest. > >Currently it's a Jenkins Job [1] that notices a change in the upstream >repo, and deploys it to the puppetmaster. Should be trivial enough to >make the changes you want there - adding a git submodule to the infra >repo might "just work", looking at it. > >[1] http://ci.theforeman.org/job/deploy_puppet/

>
>>
>>> If a separate repository, would it not be as simple as adding a git
>>>> clone?
>>>>
>>>
>>> Possibly, I don't have that much insight into the entire deployment
>>> flow. Hence my preference of making small changes since our backlog is
>>> big enough already. Of course if you're willing to take it on then be my
>>> guest.
>>>
>>
>> Currently it's a Jenkins Job [1] that notices a change in the upstream
>> repo, and deploys it to the puppetmaster. Should be trivial enough to
>> make the changes you want there - adding a git submodule to the infra
>> repo might "just work", looking at it.
>>
>> [1] http://ci.theforeman.org/job/deploy_puppet/
>>
>
> A git submodule might work - we wouldn't get automatic updates thus
> doubling the work though. If you mean adding a separate repository to the
> config then that could work well.

Another option is we just move this out of puppet all together and a have a
job that runs JJB whenever the repository updates to update the jobs
(job-ception).

··· On Thu, Nov 9, 2017 at 9:46 AM, Ewoud Kohl van Wijngaarden < ewoud@kohlvanwijngaarden.nl> wrote: > On Thu, Nov 09, 2017 at 02:37:10PM +0000, Greg Sutcliffe wrote: >> On 09/11/17 14:25, Ewoud Kohl van Wijngaarden wrote:


You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


Eric D. Helms
Red Hat Engineering