I am beginning to look at updating some of our test infrastructure by
re-writing Jenkins jobs into the pipeline plugin [1]. This is a new style,
with a different way of both writing and thinking about how jobs are
crafted. I've started this work by attempting to write jobs for both
Foreman and Katello [2].
The current test_develop job for Foreman (which runs after pull requests
are merged) is a 4x3 matrix resulting in 12 different configurations
running. They are:
We pick the most widely used Ruby version and test mysql with that
This would effectively reduce the number of test runs in the matrix to 5
which should in theory increase throughput of testing and keep things
focused on the most important pieces to test. Further, sqlite3 is not a
production database so I feel it not worth the resources (but it would only
add one more job to keep it). I also don't see how Ruby version should
affect database choice and thus find no reason to run the full matrix
across all rubies for Mysql.
From what I think I know, of the Rubies:
2.2 – used in RPM production
2.1, 2.3, 2.4 – used by Debian production
I think, Jenkins pipelines are a good thing and can make the Job
configuration a lot easier. I'd like to to pitch some ideas. First of
all I would propose putting the Jenkinsfile(s) in the repos alongside
the code so it's more visible what happens and the job configuration
follows the code lifecycle.
I've also grown fond of the Docker Plugin for Jenkins. With that you can
easily run tests under different ruby versions. I found that easier to
setup then rvm. If you'd like, I can share the code snippets. But it's
rather trivial to connfigure.
You could also try to put the databases in a container. While I usually
don't recommend this, it might be a good architectural choice for a CI
system as you don't really need to persist data. It could also greatly
decouple the jobs from the jenkins system. But first things first.
One more thing: If we have more capacity, we could start replacing
HoundCI by our own linting again.
Timo
···
Am 23.08.17 um 01:44 schrieb Eric D Helms:
I would like to propose the following:
We drop sqlite3 entirely
We test all rubies on postgresql only
We pick the most widely used Ruby version and test mysql with that
> I am beginning to look at updating some of our test infrastructure by
> re-writing Jenkins jobs into the pipeline plugin [1]. This is a new style,
> with a different way of both writing and thinking about how jobs are
> crafted. I've started this work by attempting to write jobs for both
> Foreman and Katello [2].
>
> The current test_develop job for Foreman (which runs after pull requests
> are merged) is a 4x3 matrix resulting in 12 different configurations
> running. They are:
>
> ruby: 2.1, 2.2, 2.3, 3.4
> databases: mysql, postgresql, sqlite3
>
> I would like to propose the following:
>
> 1) We drop sqlite3 entirely
>
Having an option to use sqlite3 is very convenient for small patch
devel setup, I would vote to keep it at least for one ruby version
– Ivan
···
On Wed, 23 Aug 2017 at 01:44, Eric D Helms wrote:
We test all rubies on postgresql only
We pick the most widely used Ruby version and test mysql with that
This would effectively reduce the number of test runs in the matrix to 5
which should in theory increase throughput of testing and keep things
focused on the most important pieces to test. Further, sqlite3 is not a
production database so I feel it not worth the resources (but it would only
add one more job to keep it). I also don’t see how Ruby version should
affect database choice and thus find no reason to run the full matrix
across all rubies for Mysql.
From what I think I know, of the Rubies:
2.2 – used in RPM production
2.1, 2.3, 2.4 – used by Debian production
After 1.16 branching, we will be dropping Ruby 2.1 which is only needed for
Debian Jessie iirc.
···
On Wed, Aug 23, 2017 at 2:44 AM, Eric D Helms wrote:
I am beginning to look at updating some of our test infrastructure by
re-writing Jenkins jobs into the pipeline plugin [1]. This is a new style,
with a different way of both writing and thinking about how jobs are
crafted. I’ve started this work by attempting to write jobs for both
Foreman and Katello [2].
The current test_develop job for Foreman (which runs after pull requests
are merged) is a 4x3 matrix resulting in 12 different configurations
running. They are:
We pick the most widely used Ruby version and test mysql with that
This would effectively reduce the number of test runs in the matrix to 5
which should in theory increase throughput of testing and keep things
focused on the most important pieces to test. Further, sqlite3 is not a
production database so I feel it not worth the resources (but it would only
add one more job to keep it). I also don’t see how Ruby version should
affect database choice and thus find no reason to run the full matrix
across all rubies for Mysql.
From what I think I know, of the Rubies:
2.2 – used in RPM production
2.1, 2.3, 2.4 – used by Debian production
> I am beginning to look at updating some of our test infrastructure by
> re-writing Jenkins jobs into the pipeline plugin [1]. This is a new style,
> with a different way of both writing and thinking about how jobs are
> crafted. I've started this work by attempting to write jobs for both
> Foreman and Katello [2].
+1!
> ruby: 2.1, 2.2, 2.3, 3.4
> databases: mysql, postgresql, sqlite3
>
> I would like to propose the following:
>
> 1) We drop sqlite3 entirely
At least one Ruby version should get tested with sqlite3, as almost all
dev setups will be sqlite and that's making it easy to reproduce test
failures.
> 3) We pick the most widely used Ruby version and test mysql with that
Sounds OK.
> 2.2 – used in RPM production
This will need to get updated for Rails 5 anyway.
> 2.1, 2.3, 2.4 – used by Debian production
2.1 is Debian/jessie, which will be dropped with Rails 5. 2.4 is not
used in any Debian or Ubuntu version and it's highly likely that
Debian/unstable (and thus the next Ubuntu LTS 18.04) will go from 2.3 to
2.5 directly.
At the end with Rails 5 only Ruby 2.3 and 2.4 would stay and 2.5 get
added in Q1/2018. Depending on the RPM situation 2.4 could get dropped
theroretically again in Q2/2018.
Regards
···
On Tue, Aug 22, 2017 at 07:44:42PM -0400, Eric D Helms wrote:
--
Michael Moll
I'd like to propose additional change for plugins - to only run core
tests for one Ruby version/db as well. For all other version/db
combinations we would only run plugin test suite and not core.
LZ
···
On Wed, Aug 23, 2017 at 1:44 AM, Eric D Helms wrote:
> I am beginning to look at updating some of our test infrastructure by
> re-writing Jenkins jobs into the pipeline plugin [1]. This is a new style,
> with a different way of both writing and thinking about how jobs are
> crafted. I've started this work by attempting to write jobs for both Foreman
> and Katello [2].
>
> The current test_develop job for Foreman (which runs after pull requests are
> merged) is a 4x3 matrix resulting in 12 different configurations running.
> They are:
>
> ruby: 2.1, 2.2, 2.3, 3.4
> databases: mysql, postgresql, sqlite3
>
> I would like to propose the following:
>
> 1) We drop sqlite3 entirely
> 2) We test all rubies on postgresql only
> 3) We pick the most widely used Ruby version and test mysql with that
>
> This would effectively reduce the number of test runs in the matrix to 5
> which should in theory increase throughput of testing and keep things
> focused on the most important pieces to test. Further, sqlite3 is not a
> production database so I feel it not worth the resources (but it would only
> add one more job to keep it). I also don't see how Ruby version should
> affect database choice and thus find no reason to run the full matrix across
> all rubies for Mysql.
>
> From what I think I know, of the Rubies:
>
> 2.2 -- used in RPM production
> 2.1, 2.3, 2.4 -- used by Debian production
>
> [1] https://jenkins.io/doc/book/pipeline/
> [2] https://github.com/theforeman/foreman-infra/pull/321
>
> --
> Eric D. Helms
> Red Hat Engineering
>
> --
> You received this message because you are subscribed to the Google Groups
> "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
I am beginning to look at updating some of our test infrastructure by
re-writing Jenkins jobs into the pipeline plugin [1]. This is a new style,
with a different way of both writing and thinking about how jobs are
crafted. I’ve started this work by attempting to write jobs for both
Foreman and Katello [2].
The current test_develop job for Foreman (which runs after pull requests
are merged) is a 4x3 matrix resulting in 12 different configurations
running. They are:
Having an option to use sqlite3 is very convenient for small patch
devel setup, I would vote to keep it at least for one ruby version
– Ivan
We test all rubies on postgresql only
We pick the most widely used Ruby version and test mysql with that
This would effectively reduce the number of test runs in the matrix to 5
which should in theory increase throughput of testing and keep things
focused on the most important pieces to test. Further, sqlite3 is not a
production database so I feel it not worth the resources (but it would only
add one more job to keep it). I also don’t see how Ruby version should
affect database choice and thus find no reason to run the full matrix
across all rubies for Mysql.
From what I think I know, of the Rubies:
2.2 – used in RPM production
2.1, 2.3, 2.4 – used by Debian production
There are a few different points to reply to, hopefully I did not miss one.
For the base topic, based on feedback and target for post 1.16, I think we
end up with:
2.2 - postgres
2.3 - postgres
2.4 - postgres
2.2 - mysql
2.2 - sqlite3
Jenkinsfile in Repos
I think there are valid points and reasons for why this could be a good
idea. However, I lean more towards this meaning a later goal and not an
initial target. This would provide a more Travis like experience. I think
to achieve this we need enough stability in both our job definitions, but
also in the expectations that our Jenkins would be providing for plugins
and other projects. If we want to share common code to make writing of the
jobs easier, we need to have that common code stabilized or else we end up
having to update multiple repositories every time we make a change. Thus, I
would put this on the back burner as a goal when we stabilize. The
centralized infra repository allows easier job definitions, and code
sharing and development.
Docker
I would love to get to this point compared to how RVM setups work. I think
this should be a second pass update that we make across the board after we
make the first set of jobs given that we already know the slaves are
configured and work with the RVM and database setups. I think we need to do
some up front gathering and testing of information for things like:
a) what all do we need containers of? (versions of ruby, OSes, databases,
puppet availability etc.)
b) which set of slaves can run docker? and label them
c) what configuration updates do we need to make to slaves to run docker
on them
I think this can be done in parallel to my efforts if someone would like to
take it on. We can add it to the wiki page as an item.
Hound Dropping
This probably deserves it's own thread whether we drop Hound in favor of
using Jenkins to do the linting.
Eric
···
On Wed, Aug 23, 2017 at 5:05 AM, Michael Moll wrote:
Hi,
On Tue, Aug 22, 2017 at 07:44:42PM -0400, Eric D Helms wrote:
I am beginning to look at updating some of our test infrastructure by
re-writing Jenkins jobs into the pipeline plugin [1]. This is a new
style,
with a different way of both writing and thinking about how jobs are
crafted. I’ve started this work by attempting to write jobs for both
Foreman and Katello [2].
At least one Ruby version should get tested with sqlite3, as almost all
dev setups will be sqlite and that’s making it easy to reproduce test
failures.
We pick the most widely used Ruby version and test mysql with that
Sounds OK.
2.2 – used in RPM production
This will need to get updated for Rails 5 anyway.
2.1, 2.3, 2.4 – used by Debian production
2.1 is Debian/jessie, which will be dropped with Rails 5. 2.4 is not
used in any Debian or Ubuntu version and it’s highly likely that
Debian/unstable (and thus the next Ubuntu LTS 18.04) will go from 2.3 to
2.5 directly.
At the end with Rails 5 only Ruby 2.3 and 2.4 would stay and 2.5 get
added in Q1/2018. Depending on the RPM situation 2.4 could get dropped
theroretically again in Q2/2018.
> There are a few different points to reply to, hopefully I did not miss
> one. For the base topic, based on feedback and target for post 1.16, I
> think we end up with:
>
> * 2.2 - postgres
> * 2.3 - postgres
> * 2.4 - postgres
> * 2.2 - mysql
> * 2.2 - sqlite3
>
This selection works for me
···
On Wed, 23 Aug 2017 at 15:10, Eric D Helms wrote:
Jenkinsfile in Repos
I think there are valid points and reasons for why this could be a good
idea. However, I lean more towards this meaning a later goal and not an
initial target. This would provide a more Travis like experience. I think
to achieve this we need enough stability in both our job definitions, but
also in the expectations that our Jenkins would be providing for plugins
and other projects. If we want to share common code to make writing of the
jobs easier, we need to have that common code stabilized or else we end up
having to update multiple repositories every time we make a change. Thus, I
would put this on the back burner as a goal when we stabilize. The
centralized infra repository allows easier job definitions, and code
sharing and development.
Docker
I would love to get to this point compared to how RVM setups work. I think
this should be a second pass update that we make across the board after we
make the first set of jobs given that we already know the slaves are
configured and work with the RVM and database setups. I think we need to do
some up front gathering and testing of information for things like:
a) what all do we need containers of? (versions of ruby, OSes, databases,
puppet availability etc.)
b) which set of slaves can run docker? and label them
c) what configuration updates do we need to make to slaves to run docker
on them
I think this can be done in parallel to my efforts if someone would like
to take it on. We can add it to the wiki page as an item.
Hound Dropping
This probably deserves it’s own thread whether we drop Hound in favor of
using Jenkins to do the linting.
On Tue, Aug 22, 2017 at 07:44:42PM -0400, Eric D Helms wrote:
I am beginning to look at updating some of our test infrastructure by
re-writing Jenkins jobs into the pipeline plugin [1]. This is a new
style,
with a different way of both writing and thinking about how jobs are
crafted. I’ve started this work by attempting to write jobs for both
Foreman and Katello [2].
At least one Ruby version should get tested with sqlite3, as almost all
dev setups will be sqlite and that’s making it easy to reproduce test
failures.
We pick the most widely used Ruby version and test mysql with that
Sounds OK.
2.2 – used in RPM production
This will need to get updated for Rails 5 anyway.
2.1, 2.3, 2.4 – used by Debian production
2.1 is Debian/jessie, which will be dropped with Rails 5. 2.4 is not
used in any Debian or Ubuntu version and it’s highly likely that
Debian/unstable (and thus the next Ubuntu LTS 18.04) will go from 2.3 to
2.5 directly.
At the end with Rails 5 only Ruby 2.3 and 2.4 would stay and 2.5 get
added in Q1/2018. Depending on the RPM situation 2.4 could get dropped
theroretically again in Q2/2018.
···
On Wed, 30 Aug 2017 at 11:18, Lukas Zapletal wrote:
Can we do similar for test_plugin_matrix?
I’d like to propose additional change for plugins - to only run core
tests for one Ruby version/db as well. For all other version/db
combinations we would only run plugin test suite and not core.
I am beginning to look at updating some of our test infrastructure by
re-writing Jenkins jobs into the pipeline plugin [1]. This is a new
style,
with a different way of both writing and thinking about how jobs are
crafted. I’ve started this work by attempting to write jobs for both
Foreman
and Katello [2].
The current test_develop job for Foreman (which runs after pull requests
are
merged) is a 4x3 matrix resulting in 12 different configurations running.
They are:
We pick the most widely used Ruby version and test mysql with that
This would effectively reduce the number of test runs in the matrix to 5
which should in theory increase throughput of testing and keep things
focused on the most important pieces to test. Further, sqlite3 is not a
production database so I feel it not worth the resources (but it would
only
add one more job to keep it). I also don’t see how Ruby version should
affect database choice and thus find no reason to run the full matrix
across
all rubies for Mysql.
From what I think I know, of the Rubies:
2.2 – used in RPM production
2.1, 2.3, 2.4 – used by Debian production
On středa 30. srpna 2017 12:36:01 CEST Ivan Necas wrote:
+1
– Ivan
On Wed, 30 Aug 2017 at 11:18, Lukas Zapletal lzap@redhat.com wrote:
Can we do similar for test_plugin_matrix?
I’d like to propose additional change for plugins - to only run core
tests for one Ruby version/db as well. For all other version/db
combinations we would only run plugin test suite and not core.
LZ
On Wed, Aug 23, 2017 at 1:44 AM, Eric D Helms ericdhelms@gmail.com > > > > wrote:
I am beginning to look at updating some of our test infrastructure by
re-writing Jenkins jobs into the pipeline plugin [1]. This is a new
style,
with a different way of both writing and thinking about how jobs are
crafted. I’ve started this work by attempting to write jobs for both
Foreman
and Katello [2].
The current test_develop job for Foreman (which runs after pull requests
are
merged) is a 4x3 matrix resulting in 12 different configurations
running.
We pick the most widely used Ruby version and test mysql with that
This would effectively reduce the number of test runs in the matrix to 5
which should in theory increase throughput of testing and keep things
focused on the most important pieces to test. Further, sqlite3 is not a
production database so I feel it not worth the resources (but it would
only
add one more job to keep it). I also don’t see how Ruby version should
affect database choice and thus find no reason to run the full matrix
across
all rubies for Mysql.
From what I think I know, of the Rubies:
2.2 – used in RPM production
2.1, 2.3, 2.4 – used by Debian production
>
>>
>> There are a few different points to reply to, hopefully I did not miss
>> one. For the base topic, based on feedback and target for post 1.16, I think
>> we end up with:
>>
>> * 2.2 - postgres
>> * 2.3 - postgres
>> * 2.4 - postgres
>> * 2.2 - mysql
>> * 2.2 - sqlite3
>
+1 that should provide enough coverage
···
On Wed, Aug 23, 2017 at 5:37 PM, Ivan Necas wrote:
> On Wed, 23 Aug 2017 at 15:10, Eric D Helms wrote:
This selection works for me
Jenkinsfile in Repos
I think there are valid points and reasons for why this could be a good
idea. However, I lean more towards this meaning a later goal and not an
initial target. This would provide a more Travis like experience. I think to
achieve this we need enough stability in both our job definitions, but also
in the expectations that our Jenkins would be providing for plugins and
other projects. If we want to share common code to make writing of the jobs
easier, we need to have that common code stabilized or else we end up having
to update multiple repositories every time we make a change. Thus, I would
put this on the back burner as a goal when we stabilize. The centralized
infra repository allows easier job definitions, and code sharing and
development.
Docker
I would love to get to this point compared to how RVM setups work. I think
this should be a second pass update that we make across the board after we
make the first set of jobs given that we already know the slaves are
configured and work with the RVM and database setups. I think we need to do
some up front gathering and testing of information for things like:
a) what all do we need containers of? (versions of ruby, OSes, databases,
puppet availability etc.)
b) which set of slaves can run docker? and label them
c) what configuration updates do we need to make to slaves to run docker
on them
I think this can be done in parallel to my efforts if someone would like
to take it on. We can add it to the wiki page as an item.
Hound Dropping
This probably deserves it’s own thread whether we drop Hound in favor of
using Jenkins to do the linting.
On Tue, Aug 22, 2017 at 07:44:42PM -0400, Eric D Helms wrote:
I am beginning to look at updating some of our test infrastructure by
re-writing Jenkins jobs into the pipeline plugin [1]. This is a new
style,
with a different way of both writing and thinking about how jobs are
crafted. I’ve started this work by attempting to write jobs for both
Foreman and Katello [2].
At least one Ruby version should get tested with sqlite3, as almost all
dev setups will be sqlite and that’s making it easy to reproduce test
failures.
We pick the most widely used Ruby version and test mysql with that
Sounds OK.
2.2 – used in RPM production
This will need to get updated for Rails 5 anyway.
2.1, 2.3, 2.4 – used by Debian production
2.1 is Debian/jessie, which will be dropped with Rails 5. 2.4 is not
used in any Debian or Ubuntu version and it’s highly likely that
Debian/unstable (and thus the next Ubuntu LTS 18.04) will go from 2.3 to
2.5 directly.
At the end with Rails 5 only Ruby 2.3 and 2.4 would stay and 2.5 get
added in Q1/2018. Depending on the RPM situation 2.4 could get dropped
theroretically again in Q2/2018.
Having an option to use sqlite3 is very convenient for small patch