>For example, setting the release version on a content host. This value is stored in the backend system (candlepin), so while I can stub out the calls to get/set it, that gives no indication whether candlepin >is saving or returning it as expected. I've only tested the thin veneer layer in the katello model.
But that's the purpose of unit testing, speed. Candlepin should have its own tests that test the method you are calling. End to end testing usually takes much longer than unit tests (even Katello unit tests) so it's difficult to run it before checking in code. Unit tests are supposed to be a fast way of seeing if you broke anything before checking in code. I think we need both.
>Though I don't know all of the options, I prefer minitest with VCR without any stubbing. Run in record mode for speed, while perhaps having nightly builds run in live mode. No stubbing, just testing.
I'll go ahead and disagree with that. I dislike VCR and consider it a shortcut. What happens if you record the wrong thing the begin with? All calls to backend services should be mocked (from the unit test's perspective).
>In my fantasy world, QE and dev would be writing the same test suite, the end-to-end one. It's a massive duplication for dev to write a test for "name.length > 255" and then for QE to live test it in a >different test suite. Wouldn't it be better for dev to write the test against live backend services, record it, then have QE time spend writing a different test?
I think there is a difference between end to end testing and integration testing. I think what you are referring to above is integration testing. I was under the impression that QE is writing automated end to end (or functional) tests in the browser with selenium. These tests generally test a series of actions and are quite slow. I don't think it makes sense for us to run those every time we check in code. If QE is also writing tests against the API then perhaps those could be a consolidation point but I still see the need for unit testing.
Cheers,
Walden
···
----- Original Message -----
From: "Tom McKay"
To: foreman-dev@googlegroups.com
Sent: Tuesday, May 13, 2014 8:43:35 PM
Subject: Re: [foreman-dev] [foretello] time vs. reward for tests, worth it?
----- Original Message -----
Tom,
I think it would be helpful if you could:
A reasonable request, though it detracts from the raw ranting nature of my original post.
- Provide specific examples of the issues you are describing
For example, setting the release version on a content host. This value is stored in the backend system (candlepin), so while I can stub out the calls to get/set it, that gives no indication whether candlepin is saving or returning it as expected. I’ve only tested the thin veneer layer in the katello model.
- Provide examples of the end to end testing you envision and how that can
replace other forms of testing
Though I don’t know all of the options, I prefer minitest with VCR without any stubbing. Run in record mode for speed, while perhaps having nightly builds run in live mode. No stubbing, just testing.
Perhaps only testing the API layer as well. That’s the interface that the CLI and UI hits and is the layer most likely to manifest user errors, I think, and would be highest value return on time invested in writing tests.
I understand that above statement doesn’t make for facile corner case testing, but honestly, given the choice I’d rather have 100% real use case coverage before testing that things blow up with labels greater than 255 characters. If we’re struggling to make an end-to-end test that fails in a certain case, what’s the chance of a customer hitting it?
- Provide information regarding how the end-to-end testing can save time
as you claim it might
In my fantasy world, QE and dev would be writing the same test suite, the end-to-end one. It’s a massive duplication for dev to write a test for “name.length > 255” and then for QE to live test it in a different test suite. Wouldn’t it be better for dev to write the test against live backend services, record it, then have QE time spend writing a different test?
Eric
On Tue, May 13, 2014 at 7:18 PM, Tom McKay thomasmckay@redhat.com wrote:
Does anyone else question the value of the katello tests? Running ‘rake
test:katello’ after doing anything moderately interesting with backend
services ends up being a multi-day chore of figuring out what needs to be
stubbed and mocked out. It is probably my least favorite task, overall, yet
testing is probably one of my highest favorite beliefs. Yeah, call my a
hypocrite.
What are we really testing when we perform some action and stub out all of
the guts of what is actually happening?
Is there not a better time vs. value investment that developers could be
making in adding tests? Are there better choices out there?
For me, I’d much prefer to be writing true end-to-end tests that excercise
how the tool is actually used.
–
@thomasmckay
–
“The leader must aim high, see big, judge widely, thus setting himself
apart form the ordinary people who debate in narrow confines.” ~ Charles De
Gaulle
“Leadership is about making others better as a result of your presence and
making sure that impact lasts in your absence.” ~ Harvard Business School
–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups
"foreman-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
–
You received this message because you are subscribed to the Google Groups “foreman-dev” group.
To unsubscribe from this group and stop receiving emails from it, send an email to foreman-dev+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
