As an experiment, I've added a new test to test_develop_pull_request
called "upgrade17" that tries to test an upgrade from 1.7-stable to your
pull request.
The idea is to catch model changes or regressions in the PR that will
break existing DB migrations, which often use real models rather than
fakes. As an example, a migration that iterates over hosts and updates
an attribute might later fail to validate the model if a new validation
requires a newer DB migration to operate.
For now it's just testing the single major version upgrade, but users
upgrading across multiple releases often hit these, so we could extend
these tests to two major releases (1.7 and 1.8) if they seem OK and we
want to make such a jump work.
It does the following:
- checks out the 1.7-stable branch
- bundle installs, db:migrates and seeds
- checks out develop and merges in the PR
- bundle updates, db:migrates and seeds
···
--
Dominic Cleal
Red Hat Engineering
> As an experiment, I've added a new test to test_develop_pull_request
> called "upgrade17" that tries to test an upgrade from 1.7-stable to your
> pull request.
Excellent!
> The idea is to catch model changes or regressions in the PR that will
> break existing DB migrations, which often use real models rather than
> fakes. As an example, a migration that iterates over hosts and updates
> an attribute might later fail to validate the model if a new validation
> requires a newer DB migration to operate.
>
> For now it's just testing the single major version upgrade, but users
> upgrading across multiple releases often hit these, so we could extend
> these tests to two major releases (1.7 and 1.8) if they seem OK and we
> want to make such a jump work.
I'm comfortable with our current model for only supporting one release
upgrading to the next. It's very nice to know that's been tested
though. Could be worth getting advance warning, via a "stable ->
nightly" migration too?
> It does the following:
> - checks out the 1.7-stable branch
> - bundle installs, db:migrates and seeds
> - checks out develop and merges in the PR
> - bundle updates, db:migrates and seeds
Is it worth adding an extra seed, maybe to add some hosts? A lot of
the failures we see are when we have data issues (although, clearly,
adding every possible data item to the db for this test is unfeasbile,
so maybe it's not worth it…)
Greg
···
On 17 February 2015 at 12:07, Dominic Cleal wrote:
I've added 1.8, so we now test:
a) 1.7-stable to develop+PR (skipping 1.8)
b) 1.8-stable to develop+PR
We can reconsider keeping the 1.7-stable to develop upgrade test if it
ever fails on a PR and we don't think it's worth maintaining.
···
On 17/02/15 12:07, Dominic Cleal wrote:
> As an experiment, I've added a new test to test_develop_pull_request
> called "upgrade17" that tries to test an upgrade from 1.7-stable to your
> pull request.
>
> The idea is to catch model changes or regressions in the PR that will
> break existing DB migrations, which often use real models rather than
> fakes. As an example, a migration that iterates over hosts and updates
> an attribute might later fail to validate the model if a new validation
> requires a newer DB migration to operate.
>
> For now it's just testing the single major version upgrade, but users
> upgrading across multiple releases often hit these, so we could extend
> these tests to two major releases (1.7 and 1.8) if they seem OK and we
> want to make such a jump work.
–
Dominic Cleal
Red Hat Engineering
>> The idea is to catch model changes or regressions in the PR that will
>> break existing DB migrations, which often use real models rather than
>> fakes. As an example, a migration that iterates over hosts and updates
>> an attribute might later fail to validate the model if a new validation
>> requires a newer DB migration to operate.
>>
>> For now it's just testing the single major version upgrade, but users
>> upgrading across multiple releases often hit these, so we could extend
>> these tests to two major releases (1.7 and 1.8) if they seem OK and we
>> want to make such a jump work.
>
> I'm comfortable with our current model for only supporting one release
> upgrading to the next.
Yeah, same. I don't know how much effort it would be to support
anything more - let's see how we go.
> It's very nice to know that's been tested
> though. Could be worth getting advance warning, via a "stable ->
> nightly" migration too?
Can do easily, though generally we should catch everything through PR
tests. I had started work a long while ago on a foreman-bats test that
would do an upgrade, that's probably the best place to next focus effort.
>> It does the following:
>> - checks out the 1.7-stable branch
>> - bundle installs, db:migrates and seeds
>> - checks out develop and merges in the PR
>> - bundle updates, db:migrates and seeds
>
> Is it worth adding an extra seed, maybe to add some hosts? A lot of
> the failures we see are when we have data issues (although, clearly,
> adding every possible data item to the db for this test is unfeasbile,
> so maybe it's not worth it…)
I'd like this, yep. Many of the migrations are no-ops if there aren't
any rows in the DB.
We'd need to get some reproducible data built on a stable release stored
somewhere which we can seed.
···
On 17/02/15 12:32, Greg Sutcliffe wrote:
> On 17 February 2015 at 12:07, Dominic Cleal wrote:
–
Dominic Cleal
Red Hat Engineering
> > As an experiment, I've added a new test to test_develop_pull_request
> > called "upgrade17" that tries to test an upgrade from 1.7-stable to your
> > pull request.
>
> Excellent!
>
> > The idea is to catch model changes or regressions in the PR that will
> > break existing DB migrations, which often use real models rather than
> > fakes. As an example, a migration that iterates over hosts and updates
> > an attribute might later fail to validate the model if a new validation
> > requires a newer DB migration to operate.
> >
> > For now it's just testing the single major version upgrade, but users
> > upgrading across multiple releases often hit these, so we could extend
> > these tests to two major releases (1.7 and 1.8) if they seem OK and we
> > want to make such a jump work.
>
>
> I'm comfortable with our current model for only supporting one release
> upgrading to the next. It's very nice to know that's been tested
> though. Could be worth getting advance warning, via a "stable ->
> nightly" migration too?
It would be nice if any plans to more fully test upgrade scenarios
included the requirement that you can run foreman-installer between
major versions. IMHO, this is a problem that that has to be solved
sooner rather than later.
Katello upgrades are going to be very painful in the near future. We're
lucky 2.0 to 2.1 worked, but that's absolutely certain not to be the
case for 2.2 or 2.3.
···
On Tue, Feb 17, 2015 at 12:32:27PM +0000, Greg Sutcliffe wrote:
> On 17 February 2015 at 12:07, Dominic Cleal wrote:
It does the following:
- checks out the 1.7-stable branch
- bundle installs, db:migrates and seeds
- checks out develop and merges in the PR
- bundle updates, db:migrates and seeds
Is it worth adding an extra seed, maybe to add some hosts? A lot of
the failures we see are when we have data issues (although, clearly,
adding every possible data item to the db for this test is unfeasbile,
so maybe it’s not worth it…)
Greg
–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
>> It's very nice to know that's been tested
>> though. Could be worth getting advance warning, via a "stable ->
>> nightly" migration too?
>
> Can do easily, though generally we should catch everything through PR
> tests. I had started work a long while ago on a foreman-bats test that
> would do an upgrade, that's probably the best place to next focus effort.
Good point, although without the seed data discussion below, some of
the validity of that is questionable . Some more general use of bats
for end-to-end is a very good idea, but re: your other mail, we'll
want to settle down our Rackspace usage first I think 
>> Is it worth adding an extra seed, maybe to add some hosts? A lot of
>> the failures we see are when we have data issues (although, clearly,
>> adding every possible data item to the db for this test is unfeasbile,
>> so maybe it's not worth it…)
>
> I'd like this, yep. Many of the migrations are no-ops if there aren't
> any rows in the DB.
>
> We'd need to get some reproducible data built on a stable release stored
> somewhere which we can seed.
I can offer the 99-local seed stuff I've been working on for personal
use (which allows me to wipe out my db several times a day, as all my
proxies, CRs, subnets, OSs etc are contained therein). With some
cleaning up, it might be useful?
Greg
···
On 17 February 2015 at 12:51, Dominic Cleal wrote:
>>> It's very nice to know that's been tested
>>> though. Could be worth getting advance warning, via a "stable ->
>>> nightly" migration too?
>>
>> Can do easily, though generally we should catch everything through PR
>> tests. I had started work a long while ago on a foreman-bats test that
>> would do an upgrade, that's probably the best place to next focus effort.
>
> Good point, although without the seed data discussion below, some of
> the validity of that is questionable . Some more general use of bats
> for end-to-end is a very good idea, but re: your other mail, we'll
> want to settle down our Rackspace usage first I think
I think it's valid, as the focus there is to test the packages and
installation as a whole. Plus after a typical installer run, you'll
have a host and some facts etc.
>>> Is it worth adding an extra seed, maybe to add some hosts? A lot of
>>> the failures we see are when we have data issues (although, clearly,
>>> adding every possible data item to the db for this test is unfeasbile,
>>> so maybe it's not worth it…)
>>
>> I'd like this, yep. Many of the migrations are no-ops if there aren't
>> any rows in the DB.
>>
>> We'd need to get some reproducible data built on a stable release stored
>> somewhere which we can seed.
>
> I can offer the 99-local seed stuff I've been working on for personal
> use (which allows me to wipe out my db several times a day, as all my
> proxies, CRs, subnets, OSs etc are contained therein). With some
> cleaning up, it might be useful?
Sure, I don't know where to store it though. Perhaps put it into a new
repo that can be cloned into the job, or possibly in foreman-infra and
deployed directly to slaves (not my favourite option).
···
On 17/02/15 12:55, Greg Sutcliffe wrote:
> On 17 February 2015 at 12:51, Dominic Cleal wrote:
–
Dominic Cleal
Red Hat Engineering
