Just wanted to check if anyone had any objections to a couple of
improvements to the Jenkins setup in ci.theforeman.org.
Firstly, and nice'n'simple, I'm thinking of adding the Extended Read
plugin[1]. This would allow us to give read-only access to the job
configurations, which is useful when sharing with other CI teams, or
discussing improvements in the job setup, without giving full edit
permissions.
Secondly, and slightly more work, I think there's value in using Job
Builder[2] which would allow us to express our jobs in YAML and thus store
them in Git. I've certainly wished I could better track the scripts used to
build the Deb packages, or easily export our Jenkins config into a test VM,
and this would seem to fit the requirement. Doubtless there are other
usecases for this (post-commit hooks? audit logs?). You can see how the
Ovirt team use it here[3].
I'm happy to do the work on it as a "nice 19" when I have time, but I just
wanted to see if anyone can see any potential pitfalls with either of these
plugins?
> Hi all,
>
> Just wanted to check if anyone had any objections to a couple of
> improvements to the Jenkins setup in ci.theforeman.org
> <http://ci.theforeman.org>.
>
> Firstly, and nice'n'simple, I'm thinking of adding the Extended Read
> plugin[1]. This would allow us to give read-only access to the job
> configurations, which is useful when sharing with other CI teams, or
> discussing improvements in the job setup, without giving full edit
> permissions.
Sure, though I'd like it if we could do a quick audit of them first.
There's obviously a lot of cruft, but I would also like to double check
if there's anything security-sensitive (unlikely).
> Secondly, and slightly more work, I think there's value in using Job
> Builder[2] which would allow us to express our jobs in YAML and thus
> store them in Git. I've certainly wished I could better track the
> scripts used to build the Deb packages, or easily export our Jenkins
> config into a test VM, and this would seem to fit the requirement.
> Doubtless there are other usecases for this (post-commit hooks? audit
> logs?). You can see how the Ovirt team use it here[3].
I'd like this, it'd fit fine into infra. A question out of curiousity,
can you create a job in Jenkins' UI and then export it, or do you
develop it entirely in YAML?
> > Hi all,
> >
> > Just wanted to check if anyone had any objections to a couple of
> > improvements to the Jenkins setup in ci.theforeman.org
> > <http://ci.theforeman.org>.
> >
> > Firstly, and nice'n'simple, I'm thinking of adding the Extended Read
> > plugin[1]. This would allow us to give read-only access to the job
> > configurations, which is useful when sharing with other CI teams, or
> > discussing improvements in the job setup, without giving full edit
> > permissions.
>
> Sure, though I'd like it if we could do a quick audit of them first.
> There's obviously a lot of cruft, but I would also like to double check
> if there's anything security-sensitive (unlikely).
>
Agreed, we can add the plugin and then audit with a test user before we add
the role to anyone else.
> > Secondly, and slightly more work, I think there's value in using Job
> > Builder[2] which would allow us to express our jobs in YAML and thus
> > store them in Git. I've certainly wished I could better track the
> > scripts used to build the Deb packages, or easily export our Jenkins
> > config into a test VM, and this would seem to fit the requirement.
> > Doubtless there are other usecases for this (post-commit hooks? audit
> > logs?). You can see how the Ovirt team use it here[3].
>
> I'd like this, it'd fit fine into infra. A question out of curiousity,
> can you create a job in Jenkins' UI and then export it, or do you
> develop it entirely in YAML?
>
I had a quick look through the docs and I don't see a way to export, which
is a shame. However, it won't mess with non-managed jobs unless you
explicitly say so (jjb update --delete-old), so we can migrate jobs one at
a time. I'll also try to test the initial setup and migration of simple
jobs on a test instance, just to ensure it doesn't hose anything unexpected.
We could actually delay Extended-Read until after JJB is done - rewriting
the jobs for JJB (which will then be published in foreman-infra, I guess)
would also count as a security audit.
Greg
···
On 30 October 2014 13:11, Dominic Cleal wrote:
> On 30/10/14 12:10, Greg Sutcliffe wrote:
> > I'd like this, it'd fit fine into infra. A question out of curiousity,
> > can you create a job in Jenkins' UI and then export it, or do you
> > develop it entirely in YAML?
+1
Will it possible to test jobs from our branches or PRs?
>>
>> > > I'd like this, it'd fit fine into infra. A question out of
>> > > curiousity,
>> > > can you create a job in Jenkins' UI and then export it, or do you
>> > > develop it entirely in YAML?
I wasn't entirely clear the answer on this but it sounded like I would
have to always develop the config in git? From my experience, that is
going to be annoying when a configuration does need to change due in
part to Jenkins environment always having caveats that locally don't
exist when trying to setup a new job or re-configure an existing one.
>>
>> +1
>>
>> Will it possible to test jobs from our branches or PRs?
>
>
> Could be tricky (how does one test the test infrastructure? :P), but if an
> updated job fails, at least we can revert the commit in Git, which we cannot
> do today.
JOJ - Jenkins on Jenkins?
···
On Fri, Oct 31, 2014 at 8:03 AM, Greg Sutcliffe wrote:
> On 31 October 2014 11:38, Lukas Zapletal wrote:
Could be tricky (how does one test the test infrastructure? :P), but if an
updated job fails, at least we can revert the commit in Git, which we
cannot do today.
···
On 31 October 2014 11:38, Lukas Zapletal wrote:
I’d like this, it’d fit fine into infra. A question out of curiousity,
can you create a job in Jenkins’ UI and then export it, or do you
develop it entirely in YAML?
+1
Will it possible to test jobs from our branches or PRs?
Not sure I follow - what does "local" mean here? The premise is to define
the jobs in a file and use a tool which logs into Jenkins to create/update
those jobs. We can arrange for Jenkins to be updated on a successful commit
(via a commit hook or similar) once all jobs are migrated, so then it
should be a case of "send PR, review, merge PR" just like our other repos.
On 31 October 2014 11:38, Lukas Zapletal lzap@redhat.com wrote:
I’d like this, it’d fit fine into infra. A question out of
curiousity,
can you create a job in Jenkins’ UI and then export it, or do you
develop it entirely in YAML?
I wasn’t entirely clear the answer on this but it sounded like I would
have to always develop the config in git? From my experience, that is
going to be annoying when a configuration does need to change due in
part to Jenkins environment always having caveats that locally don’t
exist when trying to setup a new job or re-configure an existing one.
>>
>> >>
>> >> > > I'd like this, it'd fit fine into infra. A question out of
>> >> > > curiousity,
>> >> > > can you create a job in Jenkins' UI and then export it, or do you
>> >> > > develop it entirely in YAML?
>>
>> I wasn't entirely clear the answer on this but it sounded like I would
>> have to always develop the config in git? From my experience, that is
>> going to be annoying when a configuration does need to change due in
>> part to Jenkins environment always having caveats that locally don't
>> exist when trying to setup a new job or re-configure an existing one.
>>
>
> Not sure I follow - what does "local" mean here? The premise is to define
> the jobs in a file and use a tool which logs into Jenkins to create/update
> those jobs. We can arrange for Jenkins to be updated on a successful commit
> (via a commit hook or similar) once all jobs are migrated, so then it should
> be a case of "send PR, review, merge PR" just like our other repos.
I am missing the 'test the update to the config works' step I think.
···
On Fri, Oct 31, 2014 at 11:04 AM, Greg Sutcliffe wrote:
> On 31 October 2014 12:55, Eric D Helms wrote:
>> On Fri, Oct 31, 2014 at 8:03 AM, Greg Sutcliffe >> wrote:
>> > On 31 October 2014 11:38, Lukas Zapletal wrote:
Right, yeah. Short version - I don't think we can. It would entail spinning
up a parallel Jenkins instance with all the same credentials and verifying
it can run all the jobs (some of which run very infrequently).
However, I still consider this an improvement. Currently, Jenkins is
maintained by a process of anyone with admin rights editing jobs and seeing
if they work. If they don't, we edit them some more until they do. Only in
cases of very large changes do we bother to clone the jobs to a new name
and test them more carefully. If anything breaks with a smaller update,
Jenkins has no audit trail - the user has to recall what was changed and
proceed accordingly.
With this approach we gain peer review of proposed changes (via having the
YAML config in Git), and a reliable rollback of changes (again, via Git).
I'd say that's worth going for, even if we (still) can't actually test
proposed updates. We can also add some basic syntax checking, etc, as
commit hooks.
As a side note - JJB would likely be deployed to the puppetmaster and run
from there via a cron. I don't want it to be dependant on Jenkins itself,
or you get into a cyclical situation where a change to the job config can't
be deployed because Jenkins is broken. Using a cron will mean that we can
always git-revert and have Jenkins correctly updated, even if all jobs were
broken up to that point.
Greg
···
On 31 October 2014 15:21, Eric D Helms wrote:
I am missing the ‘test the update to the config works’ step I think.
It's also worth noting that having our job config in a reusable format will
make it easy to import those jobs into the downstream CI system we use for
the product, which will help us ensure it's tested in the same way as
upstream.
I made a start on this. I've created a clean Jenkins VM based on our
infra puppet code - so it's set-up correctly, but has none of the
jobs. I then installed JJB via Pip, and set out to re-implement one of
the simplest jobs, deploy_web.
The result is here[1] - there's a lot of scaffold, since it's the
first job, but it's pretty nice. To use it, we check the jenkins_job
subdirectory out to /etc/jenkins_jobs, change the user/pw in the INI
file to a valid user with admin rights, and run
jenkins-jobs update /etc/jenkins_jobs/yaml
This produces something like:
INFO:root:Updating jobs in ['/etc/jenkins_jobs/config/yaml',
'/etc/jenkins_jobs/config/yaml/scm',
'/etc/jenkins_jobs/config/yaml/publishers',
'/etc/jenkins_jobs/config/yaml/builders',
'/etc/jenkins_jobs/config/yaml/defaults',
'/etc/jenkins_jobs/config/yaml/jobs'] ([])
INFO:jenkins_jobs.local_yaml:Including file
'scripts/gemset_cleanup.sh' from path 'scripts/gemset_cleanup.sh'
INFO:jenkins_jobs.local_yaml:Including file 'scripts/deploy_web.sh'
from path 'scripts/deploy_web.sh'
INFO:jenkins_jobs.builder:Reconfiguring jenkins job deploy_web
Looks pretty good to me. I like being able to have scripts in their
own files, and embed the yaml together, and I like how things like
gemset_cleanup.sh can become fully reusable across many jobs.
Next tasks are to do the other two simple deploy_* jobs, and then
start to try one of the more complex matrix jobs.
I finally got back to this after needing a break from other stuff on
Friday. I've updated the branch[1] to show the complete build pipeline
for test_proxy_develop (well, the deb half, anyway). There are a few
minor niggles I couldn;t figure out how to enable (mainly, the
"regularly reschedule failed builds" option, which doesn't seem to
have been added to JJB yet).
It's looking pretty nice, and there's definitely more we can do to DRY
up the use of things like parameters, SCM, sane defaults, and bash
script snippets. There's also an analogous "jenkins-view-builder"
which would allow us to express things like the pipeline views in
yaml, but a quick test let to code bugs, so I've stopped looking at
that for now.
Dominic, are we still wanting to implement this? What's the next step?
I can easily submit a PR, and possibly some screenshots from my test
instance showing the finished jobs?
> Update:
>
> I finally got back to this after needing a break from other stuff on
> Friday. I've updated the branch[1] to show the complete build pipeline
> for test_proxy_develop (well, the deb half, anyway). There are a few
> minor niggles I couldn;t figure out how to enable (mainly, the
> "regularly reschedule failed builds" option, which doesn't seem to
> have been added to JJB yet).
Does that mean the setting will be unset if we run it against a job that
has it ticked?
> It's looking pretty nice, and there's definitely more we can do to DRY
> up the use of things like parameters, SCM, sane defaults, and bash
> script snippets. There's also an analogous "jenkins-view-builder"
> which would allow us to express things like the pipeline views in
> yaml, but a quick test let to code bugs, so I've stopped looking at
> that for now.
>
> Dominic, are we still wanting to implement this? What's the next step?
> I can easily submit a PR, and possibly some screenshots from my test
> instance showing the finished jobs?
Yeah, PR sounds fine. We have a plugin to view job config XML diffs, so
if it does something bad we should be able to revert it (plus have
backups anyway).
Once we've got a few examples merged in there then we can continue
migrating and adding to it.
>> Update:
>>
>> I finally got back to this after needing a break from other stuff on
>> Friday. I've updated the branch[1] to show the complete build pipeline
>> for test_proxy_develop (well, the deb half, anyway). There are a few
>> minor niggles I couldn;t figure out how to enable (mainly, the
>> "regularly reschedule failed builds" option, which doesn't seem to
>> have been added to JJB yet).
>
> Does that mean the setting will be unset if we run it against a job that
> has it ticked?
A quick test with Naginator (the plugin that now provides this
functionality, and also isn't currently included in JJB) seems to show
that the extra post-build step of "Retry Failed Builds" is removed
next time JJB is run. That's a shame, but [1] seems to show extending
JJB is easy enough (even for my limited Python skills). We should be
able to get what we need, when necessary.
>> It's looking pretty nice, and there's definitely more we can do to DRY
>> up the use of things like parameters, SCM, sane defaults, and bash
>> script snippets. There's also an analogous "jenkins-view-builder"
>> which would allow us to express things like the pipeline views in
>> yaml, but a quick test let to code bugs, so I've stopped looking at
>> that for now.
>>
>> Dominic, are we still wanting to implement this? What's the next step?
>> I can easily submit a PR, and possibly some screenshots from my test
>> instance showing the finished jobs?
>
> Yeah, PR sounds fine. We have a plugin to view job config XML diffs, so
> if it does something bad we should be able to revert it (plus have
> backups anyway).
Excellent, I'll prep up a PR that includes setting up JJB, but without
a link between the git commits and running JJB, for now. Then we can
manually inspect and test before continuing.
> Once we've got a few examples merged in there then we can continue
> migrating and adding to it.
Agreed. At the least, we should finish up the proxy pipeline for RPM
and see how that goes.
Greg
···
On 27 January 2015 at 08:18, Dominic Cleal wrote:
> On 26/01/15 17:15, Greg Sutcliffe wrote:
···
On Wed, Jan 28, 2015 at 10:03:29AM +0000, Greg Sutcliffe wrote:
> On 27 January 2015 at 08:18, Dominic Cleal wrote:
> > On 26/01/15 17:15, Greg Sutcliffe wrote:
> >> Update:
> >>
> >> I finally got back to this after needing a break from other stuff on
> >> Friday. I've updated the branch[1] to show the complete build pipeline
> >> for test_proxy_develop (well, the deb half, anyway). There are a few
> >> minor niggles I couldn;t figure out how to enable (mainly, the
> >> "regularly reschedule failed builds" option, which doesn't seem to
> >> have been added to JJB yet).
> >
> > Does that mean the setting will be unset if we run it against a job that
> > has it ticked?
>
> A quick test with Naginator (the plugin that now provides this
> functionality, and also isn't currently included in JJB) seems to show
> that the extra post-build step of "Retry Failed Builds" is removed
> next time JJB is run. That's a shame, but [1] seems to show extending
> JJB is easy enough (even for my limited Python skills). We should be
> able to get what we need, when necessary.
