This most recent failure has me thinking again about the difficulties of being a plugin maintainer
for Foreman.
Would it be possible to get a standard plugin in Foreman's test matrix for PR? Perhaps tasks is a good
candidate. Not on all databases/rubies, but something might help gain awareness of the impact of
certain changes on plugins. Katello is there, but it doesn't run all (any?) of Foreman's tests.
Along side that, we should be tracking what plugin maintainers need to do to update from one release
to the next.
It really is a recurring problem for me, that Salt is constantly broken due to incompatibilities
introduced in Foreman. We do a poor job of tracking them. Maybe if I'm lucky someone is nice
enough to send a mail to foreman-dev. Or if the planets align, someone opens a PR to foreman_salt
and fixes it. But more often than not, my project just goes red and I have to figure out why.
But this is just a side thing, and I rarely get to immediately look at failures. They end up piling
up.
Very frustrating, as when I do find time to work on it, it's not to do something cool, but just
to figure out what changed in core and how to fix it in the plugin :-\
The only think I can tell so far is that I can't reproduce this
locally and I have no
idea on what might be different from my vs. jenkins environment that causes this
to happen.
– Ivan
···
On Mon, Oct 3, 2016 at 10:28 PM, Stephen Benjamin wrote:
> Tasks, salt and a bunch of plugins are failing tests, has anyone has a chance to look into it? e.g.:
>
> http://ci.theforeman.org/job/test_plugin_matrix/2015/#showFailuresLink
>
> This most recent failure has me thinking again about the difficulties of being a plugin maintainer
> for Foreman.
>
> Would it be possible to get a standard plugin in Foreman's test matrix for PR? Perhaps tasks is a good
> candidate. Not on all databases/rubies, but something might help gain awareness of the impact of
> certain changes on plugins. Katello is there, but it doesn't run all (any?) of Foreman's tests.
>
> Along side that, we should be tracking what plugin maintainers need to do to update from one release
> to the next.
>
> It really is a recurring problem for me, that Salt is constantly broken due to incompatibilities
> introduced in Foreman. We do a poor job of tracking them. Maybe if I'm lucky someone is nice
> enough to send a mail to foreman-dev. Or if the planets align, someone opens a PR to foreman_salt
> and fixes it. But more often than not, my project just goes red and I have to figure out why.
> But this is just a side thing, and I rarely get to immediately look at failures. They end up piling
> up.
>
> Very frustrating, as when I do find time to work on it, it's not to do something cool, but just
> to figure out what changed in core and how to fix it in the plugin :-\
>
>
> - Stephen
>
> --
> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Perhaps the change is the access permissions test was in lib, and not
it's part of the unit
Still no idea why it happens in jenkins only
– Ivan
···
On Mon, Oct 3, 2016 at 10:35 PM, Ivan Necas wrote:
> The only think I can tell so far is that I can't reproduce this
> locally and I have no
> idea on what might be different from my vs. jenkins environment that causes this
> to happen.
>
> -- Ivan
>
> On Mon, Oct 3, 2016 at 10:28 PM, Stephen Benjamin wrote:
>> Tasks, salt and a bunch of plugins are failing tests, has anyone has a chance to look into it? e.g.:
>>
>> http://ci.theforeman.org/job/test_plugin_matrix/2015/#showFailuresLink
>>
>> This most recent failure has me thinking again about the difficulties of being a plugin maintainer
>> for Foreman.
>>
>> Would it be possible to get a standard plugin in Foreman's test matrix for PR? Perhaps tasks is a good
>> candidate. Not on all databases/rubies, but something might help gain awareness of the impact of
>> certain changes on plugins. Katello is there, but it doesn't run all (any?) of Foreman's tests.
>>
>> Along side that, we should be tracking what plugin maintainers need to do to update from one release
>> to the next.
>>
>> It really is a recurring problem for me, that Salt is constantly broken due to incompatibilities
>> introduced in Foreman. We do a poor job of tracking them. Maybe if I'm lucky someone is nice
>> enough to send a mail to foreman-dev. Or if the planets align, someone opens a PR to foreman_salt
>> and fixes it. But more often than not, my project just goes red and I have to figure out why.
>> But this is just a side thing, and I rarely get to immediately look at failures. They end up piling
>> up.
>>
>> Very frustrating, as when I do find time to work on it, it's not to do something cool, but just
>> to figure out what changed in core and how to fix it in the plugin :-\
>>
>>
>> - Stephen
>>
>> --
>> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
>> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
Remove your test db, then try RAILS_ENV=test ruby -I'test'
test/unit/foreman/access_permissions_test.rb and you'll see it
happening. I think it's just because tests are loaded earlier than
permissions from plugins now.
···
On Tue, Oct 4, 2016 at 12:39 PM, Ivan Necas wrote:
> So far, I can tell this PR
> https://github.com/theforeman/foreman/pull/3843 introduced the
> behaviour
>
> as it's green on
> http://ci.theforeman.org/job/test_plugin_matrix/2059/database=mysql,ruby=2.3,slave=fast/
> and red on following commit
> http://ci.theforeman.org/job/test_plugin_matrix/2060/database=mysql,ruby=2.3,slave=fast/
>
> Perhaps the change is the access permissions test was in lib, and not
> it's part of the unit
>
> Still no idea why it happens in jenkins only
>
> -- Ivan
>
> On Mon, Oct 3, 2016 at 10:35 PM, Ivan Necas wrote:
>> The only think I can tell so far is that I can't reproduce this
>> locally and I have no
>> idea on what might be different from my vs. jenkins environment that causes this
>> to happen.
>>
>> -- Ivan
>>
>> On Mon, Oct 3, 2016 at 10:28 PM, Stephen Benjamin wrote:
>>> Tasks, salt and a bunch of plugins are failing tests, has anyone has a chance to look into it? e.g.:
>>>
>>> http://ci.theforeman.org/job/test_plugin_matrix/2015/#showFailuresLink
>>>
>>> This most recent failure has me thinking again about the difficulties of being a plugin maintainer
>>> for Foreman.
>>>
>>> Would it be possible to get a standard plugin in Foreman's test matrix for PR? Perhaps tasks is a good
>>> candidate. Not on all databases/rubies, but something might help gain awareness of the impact of
>>> certain changes on plugins. Katello is there, but it doesn't run all (any?) of Foreman's tests.
>>>
>>> Along side that, we should be tracking what plugin maintainers need to do to update from one release
>>> to the next.
>>>
>>> It really is a recurring problem for me, that Salt is constantly broken due to incompatibilities
>>> introduced in Foreman. We do a poor job of tracking them. Maybe if I'm lucky someone is nice
>>> enough to send a mail to foreman-dev. Or if the planets align, someone opens a PR to foreman_salt
>>> and fixes it. But more often than not, my project just goes red and I have to figure out why.
>>> But this is just a side thing, and I rarely get to immediately look at failures. They end up piling
>>> up.
>>>
>>> Very frustrating, as when I do find time to work on it, it's not to do something cool, but just
>>> to figure out what changed in core and how to fix it in the plugin :-\
>>>
>>>
>>> - Stephen
>>>
>>> --
>>> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
Fairly certain now it happens because of the removal of the lib task on
that PR you link. It used to run migrations twice, once on units, and
yet again on 'lib', but now 'units' runs the lib task twice.
with this patch (https://github.com/theforeman/foreman/pull/3916) and
you'll see the problem is that plugins believe there are pending
migrations. This is false though, as bundle exec rake db:migrate:status
confirms it in the script.
I'm now checking if the pending_migrations method is wrong and how can I
fix it to detect that better.
···
On 10/04, Daniel Lobato wrote:
> Remove your test db, then try RAILS_ENV=test ruby -I'test'
> test/unit/foreman/access_permissions_test.rb and you'll see it
> happening. I think it's just because tests are loaded earlier than
> permissions from plugins now.
>
> On Tue, Oct 4, 2016 at 12:39 PM, Ivan Necas wrote:
> > So far, I can tell this PR
> > https://github.com/theforeman/foreman/pull/3843 introduced the
> > behaviour
> >
> > as it's green on
> > http://ci.theforeman.org/job/test_plugin_matrix/2059/database=mysql,ruby=2.3,slave=fast/
> > and red on following commit
> > http://ci.theforeman.org/job/test_plugin_matrix/2060/database=mysql,ruby=2.3,slave=fast/
> >
> > Perhaps the change is the access permissions test was in lib, and not
> > it's part of the unit
> >
> > Still no idea why it happens in jenkins only
> >
> > -- Ivan
> >
> > On Mon, Oct 3, 2016 at 10:35 PM, Ivan Necas wrote:
> >> The only think I can tell so far is that I can't reproduce this
> >> locally and I have no
> >> idea on what might be different from my vs. jenkins environment that causes this
> >> to happen.
> >>
> >> -- Ivan
> >>
> >> On Mon, Oct 3, 2016 at 10:28 PM, Stephen Benjamin wrote:
> >>> Tasks, salt and a bunch of plugins are failing tests, has anyone has a chance to look into it? e.g.:
> >>>
> >>> http://ci.theforeman.org/job/test_plugin_matrix/2015/#showFailuresLink
> >>>
> >>> This most recent failure has me thinking again about the difficulties of being a plugin maintainer
> >>> for Foreman.
> >>>
> >>> Would it be possible to get a standard plugin in Foreman's test matrix for PR? Perhaps tasks is a good
> >>> candidate. Not on all databases/rubies, but something might help gain awareness of the impact of
> >>> certain changes on plugins. Katello is there, but it doesn't run all (any?) of Foreman's tests.
> >>>
> >>> Along side that, we should be tracking what plugin maintainers need to do to update from one release
> >>> to the next.
> >>>
> >>> It really is a recurring problem for me, that Salt is constantly broken due to incompatibilities
> >>> introduced in Foreman. We do a poor job of tracking them. Maybe if I'm lucky someone is nice
> >>> enough to send a mail to foreman-dev. Or if the planets align, someone opens a PR to foreman_salt
> >>> and fixes it. But more often than not, my project just goes red and I have to figure out why.
> >>> But this is just a side thing, and I rarely get to immediately look at failures. They end up piling
> >>> up.
> >>>
> >>> Very frustrating, as when I do find time to work on it, it's not to do something cool, but just
> >>> to figure out what changed in core and how to fix it in the plugin :-\
> >>>
> >>>
> >>> - Stephen
> >>>
> >>> --
> >>> You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> >>> To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
> >>> For more options, visit https://groups.google.com/d/optout.
> >
> > --
> > You received this message because you are subscribed to the Google Groups "foreman-dev" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
> > For more options, visit https://groups.google.com/d/optout.
>
>
>
> --
> Daniel Lobato
>
> @dlobatog
> daniellobato.me
>
> GPG: http://keys.gnupg.net/pks/lookup?op=get&search=0x7A92D6DD38D6DE30
What happens here is that db:migrate and db:migrate:status operate on the
development database and the test one remains untouched. However, running
the test for the first time automagically (and silently) migrates the test
db and the tests fail. Running the tests once more makes them pass.
$ bundle exec rake db:migrate:status RAILS_ENV=test 2>/dev/null | head
The Apipie cache is turned off. Enable it and run apipie:cache rake task to
speed up API calls.
database: foreman_test
Status Migration ID Migration Name
up 20090714132448 Create hosts
up 20090714132449 Add audits table
up 20090715143858 Create architectures
up 20090717025820 Create media
$ ruby -I test -I lib test/unit/foreman/access_permissions_test.rb # Same
results with bundle exec rake jenkins:unit
Fairly certain now it happens because of the removal of the lib task on
that PR you link. It used to run migrations twice, once on units, and
yet again on ‘lib’, but now ‘units’ runs the lib task twice.
with this patch (https://github.com/theforeman/foreman/pull/3916) and
you’ll see the problem is that plugins believe there are pending
migrations. This is false though, as bundle exec rake db:migrate:status
confirms it in the script.
I’m now checking if the pending_migrations method is wrong and how can I
fix it to detect that better.
On 10/04, Daniel Lobato wrote:
Remove your test db, then try RAILS_ENV=test ruby -I’test’
test/unit/foreman/access_permissions_test.rb and you’ll see it
happening. I think it’s just because tests are loaded earlier than
permissions from plugins now.
On Tue, Oct 4, 2016 at 12:39 PM, Ivan Necas inecas@redhat.com wrote:
Perhaps the change is the access permissions test was in lib, and not
it’s part of the unit
Still no idea why it happens in jenkins only
– Ivan
On Mon, Oct 3, 2016 at 10:35 PM, Ivan Necas inecas@redhat.com wrote:
The only think I can tell so far is that I can’t reproduce this
locally and I have no
idea on what might be different from my vs. jenkins environment that
causes this
to happen.
– Ivan
On Mon, Oct 3, 2016 at 10:28 PM, Stephen Benjamin stephen@redhat.com > wrote:
Tasks, salt and a bunch of plugins are failing tests, has anyone has
a chance to look into it? e.g.:
