There seems to be so much emphasis on testing in the Rails community.
This is just food for thought…
Kreitzberg and Shneiderman
* Testing by itself does not improve software quality. Test results
are an indicator of quality, but in and of themselves, they don’t
improve it. Trying to improve software quality by increasing the amount
of testing is like trying to lose weight by weighing yourself more
often. What you eat before you step onto the scale determines how much
you will weigh, and the software development techniques you use
determine how many errors testing will find. If you want to lose weight,
don’t buy a new scale; change your diet. If you want to improve your
software, don’t test more; develop better.
Dominic S. wrote:
There seems to be so much emphasis on testing in the Rails community.
This is just food for thought…
Kreitzberg and Shneiderman
* Testing by itself does not improve software quality.
That discusses big-bang testing, by a QA crew, at the end of an
iteration. And you can bet the iteration was longer than one week.
Rails and other “agile” projects rely on tests written by the same
engineers who write the features.
I myself don’t think that endless debugging improves quality very
much, so I add tests as I go, and if they fail unexpectedly, I have
the option to Revert back to the last state where all tests pass. Oh,
and naturally we configure a rake submit command that only commits
code to SVN if it first passes a test run.
Quality skyrockets, and the tests themselves are just part of the
system. It’s how you use the tests that count.
If you want to improve your
software, don’t test more; develop better.
Oh, if I want higher quality, I need to raise my quality. Gotcha!
–
Phlip
http://c2.com/cgi/wiki?ZeekLand ← NOT a blog!!
I’ll agree that testing is just a tool. Even the best tool will not
help if
not applied properly. Testing probably won’t make up for poor
development
skills. But, as someone who has been using rails testing for a short
time
now, I believe it to be a very valuable tool. I find that testing before
and
after each change I make has helped me catch mistakes quickly. This, (
along wth SVN), has been especially useful during long programming
sessions
when the mind starts to get dull. I’m sure as my understanding of rails
testing improves, my software quality will improve as well. I look
forward
to the journey.
-Larry
On 2/14/07, Dominic S. [email protected] wrote:
of testing is like trying to lose weight by weighing yourself more
–
Best Regards,
-Larry
“Work, work, work…there is no satisfactory alternative.”
— E.Taft Benson
On 2/15/07, Dominic S. [email protected] wrote:
often. What you eat before you step onto the scale determines how much
you will weigh, and the software development techniques you use
determine how many errors testing will find. If you want to lose weight,
don’t buy a new scale; change your diet. If you want to improve your
software, don’t test more; develop better.
Automated tests cannot prove the absence of bugs, they can only prove
their presence.
In response to the last statement of the quote above: If you want to
improve your software, refactor. If you cannot refactor your software
because you’re afraid you might break something, you don’t have enough
automated tests.
–max
Max M. wrote:
Automated tests cannot prove the absence of bugs, they can only prove
their presence.
Washing your hands before performing surgery cannot get rid of /all/
germs, so you may as well not bother! 
–
Phlip
Washing your hands before performing surgery cannot get rid of /all/
germs, so you may as well not bother!
Well said :0)
All of this is also assuming that the only benefits that we get from
testing are less bugs. There is a trend towards Test-driven
development in the rails community and once you start doing that,
testing becomes far more than just ‘finding bugs’.
Also, the original quote’s metaphor breaks down when applied to TDD.
Testing first is like saying “What weight do I want to be?” then
working out until you get to that weight - which makes far more sense :
0)
And man… ‘develop better’? - what a nerdy thing to say…
Writing tests informs the design of whatever I’m testing. I often
refactor code to make it more testable, which generally means
decreasing coupling, improving interfaces, and applying consistent
conventions. These improvements don’t just improve testability, they
improve the overally quality of my code.
More tests + intelligent refactoring == better code.
Max M. wrote:
Automated tests cannot prove the absence of bugs, they can only prove
their presence.
Washing your hands before performing surgery cannot get rid of /all/
germs, so you may as well not bother! 
That’s not what I meant at all.
There are actually software engineers out there who use the line you
quoted as an excuse not to write any tests. (I generally figured you
quoted it tongue-in-cheek, and I responded in kind, with a winky.)
Of course unit tests are incredibly
important and even terrible test coverage is better than no coverage.
But it is dangerous to assume a piece of software to be bug-free
because all the unit tests pass. A failed test is one that has just
caught a bug. A passed test is one that may just not be testing the
right edge case.
Focus on the word “assume”. Tests don’t exist in isolation, written
requirements don’t either, and version controllers don’t, and so on.
Nobody on a team who uses TDD and frequent iterations assumes that
software is life-critical-ready just because we TDDed it. If anything,
TDD adds to the metrics showing exactly how frail our software is. A
unit test that springs loose half the time is valuable because we can
still fix it, pass it, and check in. It’s also valuable as a negative
metric.
–
Phlip
http://c2.com/cgi/wiki?ZeekLand ← NOT a blog!!
On 2/16/07, Phlip [email protected] wrote:
Phlip
That’s not what I meant at all. Of course unit tests are incredibly
important and even terrible test coverage is better than no coverage.
But it is dangerous to assume a piece of software to be bug-free
because all the unit tests pass. A failed test is one that has just
caught a bug. A passed test is one that may just not be testing the
right edge case.
–max
On 2/16/07, Phlip [email protected] wrote:
There are actually software engineers out there who use the line you
quoted as an excuse not to write any tests. (I generally figured you
quoted it tongue-in-cheek, and I responded in kind, with a winky.)
I did - only I sort-of missed the winky
Nobody on a team who uses TDD and frequent iterations assumes that
software is life-critical-ready just because we TDDed it. If anything,
TDD adds to the metrics showing exactly how frail our software is. A
unit test that springs loose half the time is valuable because we can
still fix it, pass it, and check in. It’s also valuable as a negative
metric.
I’m in violent agreement here. I cannot imagine working on a software
project without great test coverage. I can also not imagine not
conducting design and code reviews. I have, on the other hand, seen
projects forego any sort of review process because they had (close to)
100% test coverage.
–max