This came up on 3rd Oct, but I think the second half of Stephen's email got
lost in the discussion of the immediate issues. To summarize:
> From: Stephen Benjamin <email@example.com>
> Date: 3 October 2016 at 21:28
> Subject: [foreman-dev] Plugin test failures
> To: Foreman <firstname.lastname@example.org>
> Would it be possible to get a standard plugin in Foreman's test matrix
for PR? Perhaps tasks is a good candidate. Not on all databases/rubies,
but something might help gain awareness of the impact of certain changes on
plugins. Katello is there, but it doesn't run all (any?) of Foreman's tests.
I think this is a good plan. I'd be happy to see tasks or maybe salt added
to the test matrix in the short term. I also think it might make sense to
have a dedicated "fake" plugin that touches a whole ton of stuff just for
the sake of knowing if we've broken it. We could look to that for the
> Along side that, we should be tracking what plugin maintainers need to do
to update from one release to the next.
This is good too - I'll be honest and say that my plugins don't seem to
break as often as yours (just luck I guess) but it is useful when people
come tell me what needs updating.
I might be wrong, but I don't think we merge breaking changes after
freeze-date, so it should be possible to track this and then tell authors
about the changes when the first RCs go out - but first we need the "fake"
plugin that touches tons of stuff, so we know what the diffs look like (we
could even just send a list of commits to the fake plugin, showing how to
change the methods).
I can see the main concerns being:
- Test resources - do we have enough to run this on every PR? Do we want
to, or should it be less frequent?
- Should it block merges, or be advisory only? If yes, do PR authors need
to prepare patches for the plugin when they break a plugin?
I'm happy to write this up as an RFC if others think its worth having a
more formal discussion, or if one of the people with access wants to give
running foreman_tasks/salt on PRs a trial, that's cool too.