Usergroup controller test failures

Development
1

Hi,

On foreman-tasks, we're seeing failures on postgres and mysql for the Usergroup controller: http://ci.theforeman.org/job/test_plugin_matrix/921/database=mysql,ruby=2.1,slave=fast/console

I've run the tests several times, and the failures seem consistent, and not transient like some of the other things we've seen post-rails 4. Locally works fine, any ideas?

Is it possible to get access to logs and such from jenkins? I know it's somehow possible, but I can't remember - I'd like to see the test.log, and maybe even grab the Gemfile.lock to reproduce the environment fully.

Here's the errors we see:

12:12:02 Finished in 577.323481s, 5.8044 runs/s, 13.4725 assertions/s.
12:12:02
12:12:02 1) Failure:
12:12:02 UsergroupsControllerTest::GET #edit#test_0001_assigns the found object to an instance variable [/var/lib/workspace/workspace/test_plugin_matrix/database/mysql/ruby/2.1/slave/fast/foreman/test/functional/shared/basic_rest_response_test.rb:32]:
12:12:02 — expected
12:12:02 +++ actual
12:12:02 @@ -1 +1 @@
12:12:02 -#<Usergroup id: 1, name: "usergroup1", created_at: "2016-01-18 12:01:25", updated_at: "2016-01-18 12:01:25", admin: false>
12:12:02 +nil
12:12:02
12:12:02
12:12:02
12:12:02 2) Failure:
12:12:02 UsergroupsControllerTest::GET #edit#test_: UsergroupsController::GET #edit should render template edit. [/usr/local/rvm/gems/ruby-2.1.5@test_plugin_matrix-0/gems/shoulda-context-1.2.1/lib/shoulda/context/context.rb:344]:
12:12:02 expecting <"edit"> but rendering with <["common/404", "layouts/base", "layouts/application"]>
12:12:02
12:12:02
12:12:02 3) Failure:
12:12:02 UsergroupsControllerTest::GET #edit#test_: UsergroupsController::GET #edit should respond with 200. [/usr/local/rvm/gems/ruby-2.1.5@test_plugin_matrix-0/gems/shoulda-context-1.2.1/lib/shoulda/context/context.rb:344]:
12:12:02 Expected response to be a 200, but was 404
12:12:02
12:12:02 3351 runs, 7778 assertions, 3 failures, 0 errors, 0 skips

A-OK locally:

Running:

…

Finished in 19.923396s, 1.6062 runs/s, 2.0077 assertions/s.

32 runs, 40 assertions, 0 failures, 0 errors, 0 skips
Coverage report generated for Functional Tests to /home/vagrant/foreman/coverage. 7305 / 27887 LOC (26.2%) covered.

2

We don't archive the logs (disk space), but as one of these failures was
the last build, I copied it from one of the slaves - from this job in
particular:
http://ci.theforeman.org/job/test_plugin_matrix/922/database=mysql,ruby=2.0.0,slave=fast/.

The log/test.log isn't a lot of good, as I think the various different
rake tasks reinitialise the environment and truncate the log file. It
only contains http://paste.fedoraproject.org/311990/45312813.

The Gemfile.lock is at http://paste.fedoraproject.org/311991/31281591.

··· On 18/01/16 14:34, Stephen Benjamin wrote: > Hi, > > On foreman-tasks, we're seeing failures on postgres and mysql for the Usergroup controller: http://ci.theforeman.org/job/test_plugin_matrix/921/database=mysql,ruby=2.1,slave=fast/console > > I've run the tests several times, and the failures seem consistent, and not transient like some of the other things we've seen post-rails 4. Locally works fine, any ideas? > > Is it possible to get access to logs and such from jenkins? I know it's somehow possible, but I can't remember - I'd like to see the test.log, and maybe even grab the Gemfile.lock to reproduce the environment fully.

–
Dominic Cleal
dominic@cleal.org

3

I should add that you can also get this through the Workspace link on
Jenkins, under the job (not the build) for a particular part of the
matrix.

It's not super-reliable as I think it just links to the workspace of the
latest/a currently executing job, so because we don't archive all of the
files, it's from whatever's running (which could be any plugin).

e.g.
http://ci.theforeman.org/job/test_plugin_matrix/database=postgresql,ruby=2.0.0,slave=fast/ws/foreman/log/

··· On 18/01/16 14:44, Dominic Cleal wrote: > On 18/01/16 14:34, Stephen Benjamin wrote: >> Hi, >> >> On foreman-tasks, we're seeing failures on postgres and mysql for the Usergroup controller: http://ci.theforeman.org/job/test_plugin_matrix/921/database=mysql,ruby=2.1,slave=fast/console >> >> I've run the tests several times, and the failures seem consistent, and not transient like some of the other things we've seen post-rails 4. Locally works fine, any ideas? >> >> Is it possible to get access to logs and such from jenkins? I know it's somehow possible, but I can't remember - I'd like to see the test.log, and maybe even grab the Gemfile.lock to reproduce the environment fully. > > We don't archive the logs (disk space), but as one of these failures was > the last build, I copied it from one of the slaves - from this job in > particular: > http://ci.theforeman.org/job/test_plugin_matrix/922/database=mysql,ruby=2.0.0,slave=fast/. > > The log/test.log isn't a lot of good, as I think the various different > rake tasks reinitialise the environment and truncate the log file. It > only contains http://paste.fedoraproject.org/311990/45312813.

–
Dominic Cleal
dominic@cleal.org

4

Thanks! Unfortunately can't seem to reproduce the failure locally :frowning: Both branches are up
to date, gem versions should be identical. Are there any notable differences in the Jenkins
environment that might affect something like this?

This is related to to the move to shoulda a few days ago:
https://github.com/theforeman/foreman/pull/2651

What I find odd is that it's hitting mysql and postgresql, but not sqlite3.

Usergroups controller is handled differently than others, why is there this Usergroup.first || FactoryGirl bit?

https://github.com/theforeman/foreman/blob/develop/test/functional/usergroups_controller_test.rb#L10

Others don't:

https://github.com/theforeman/foreman/blob/develop/test/functional/architectures_controller_test.rb#L6

I guess because of the use of fixtures - Usergroups doesn't have fixtures. Also the setup block doesn't
seem to be called before the basic_edit_test - any ideas why? Usergroup.first doesn't exist there ever, so
it is using the FactoryGirl.create in every case.

I suspect something fishy with SQL transactions here, but I cannot seem to figure out exactly what, or
why I don't see it on my local Foreman.

··· ----- Original Message ----- > From: "Dominic Cleal" > To: foreman-dev@googlegroups.com > Sent: Monday, January 18, 2016 4:33:09 PM > Subject: Re: [foreman-dev] Usergroup controller test failures > > On 18/01/16 14:44, Dominic Cleal wrote: > > On 18/01/16 14:34, Stephen Benjamin wrote: > >> Hi, > >> > >> On foreman-tasks, we're seeing failures on postgres and mysql for the > >> Usergroup controller: > >> http://ci.theforeman.org/job/test_plugin_matrix/921/database=mysql,ruby=2.1,slave=fast/console > >> > >> I've run the tests several times, and the failures seem consistent, and > >> not transient like some of the other things we've seen post-rails 4. > >> Locally works fine, any ideas? > >> > >> Is it possible to get access to logs and such from jenkins? I know it's > >> somehow possible, but I can't remember - I'd like to see the test.log, > >> and maybe even grab the Gemfile.lock to reproduce the environment fully. > > > > We don't archive the logs (disk space), but as one of these failures was > > the last build, I copied it from one of the slaves - from this job in > > particular: > > http://ci.theforeman.org/job/test_plugin_matrix/922/database=mysql,ruby=2.0.0,slave=fast/. > > > > The log/test.log isn't a lot of good, as I think the various different > > rake tasks reinitialise the environment and truncate the log file. It > > only contains http://paste.fedoraproject.org/311990/45312813. > > I should add that you can also get this through the Workspace link on > Jenkins, under the job (*not* the build) for a particular part of the > matrix. > > It's not super-reliable as I think it just links to the workspace of the > latest/a currently executing job, so because we don't archive all of the > files, it's from whatever's running (which could be any plugin). > > e.g. > http://ci.theforeman.org/job/test_plugin_matrix/database=postgresql,ruby=2.0.0,slave=fast/ws/foreman/log/ > > -- > Dominic Cleal > dominic@cleal.org > > -- > You received this message because you are subscribed to the Google Groups > "foreman-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to foreman-dev+unsubscribe@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. >
5

Transactions (and DB cleaner etc) may come into it as the FG.create call
in basic_edit_test(…) is in the class context - it shouldn't be there
IMHO. It should only be called in the context of the test method or a
setup call.

··· On 19/01/16 08:43, Stephen Benjamin wrote: > Usergroups controller is handled differently than others, why is there this `Usergroup.first || FactoryGirl` bit? > > https://github.com/theforeman/foreman/blob/develop/test/functional/usergroups_controller_test.rb#L10 > > Others don't: > > https://github.com/theforeman/foreman/blob/develop/test/functional/architectures_controller_test.rb#L6 > > I guess because of the use of fixtures - Usergroups doesn't have fixtures. Also the setup block doesn't > seem to be called before the basic_edit_test - any ideas why? Usergroup.first doesn't exist there ever, so > it is using the FactoryGirl.create in every case. > > I *suspect* something fishy with SQL transactions here, but I cannot seem to figure out exactly what, or > why I don't see it on my local Foreman.

–
Dominic Cleal
dominic@cleal.org