Playing with a Testing Library
-
I'd like to be able to express my unit tests fairly naturally, using the conditional operators built into the language. So, for example, I'd want to write:
expect(factorial(5)) == 120 expect(factorial(10)) > 10000 -
I'd like the error messages to show both the code that caused the error and the values that caused the error. So, for example, I'd want the following (incorrect) test
expect(factorial(6)) == 600to output something like
/Users/dave/tmp/tmc/blog_tests.rb:16 the code was: expect(factorial(6)) == 600, but 720 != 600and
expect(1) > 2should say
/Users/dave/Play/tmc/blog_tests.rb:11 the code was: expect(1) > 2, but 1 <= 2(Note how the expression showing the actual values negates the comparison operator to make it easier to read.)
-
I annotate my code with comments, so I'd like to be able to annotate my
tests the same way.
expect(factorial(6)) == 600 # Deliberate bad testshould produce something like
/Users/dave/tmp/tmc/blog_tests.rb:17 Deliberate bad test the code was: expect(factorial(6)) == 600, but 720 != 600Sometimes I write longer comments.
# The factorial of 6 is a special case, # because of the labor laws in Las Vegas expect(factorial(6)) == 600So the resulting errors are longer, too.
/Users/dave/Play/tmc/blog_tests.rb:21 The factorial of 6 is a special case, because of the labor laws in Las Vegas the code was: expect(factorial(6)) == 600, but 720 != 600 -
I like to be able to group my tests.
testing("positive factorials") do expect(factorial(1)) == 1 expect(factorial(2)) == 2 expect(factorial(5)) == 120 end testing("factorial of zero") do expect(factorial(0)) == 1 end testing("negative factorials") do expect(factorial(-1)) == 1 expect(factorial(-5)) == 1 end -
I like the description of the group to appear along with any individual test annotation if a test fails.
will producetesting("factorial of zero") do # this test is deliberately wrong expect(factorial(0)) == 0 end/Users/dave/Play/tmc/blog_tests.rb:31--while testing factorial of zero this test is deliberately wrong the code was: expect(factorial(0)) == 0, but 1 != 0 -
I like to have the flexibility to set up the environment for a group of tests. I also like to have the idea of a global environment which doesn't get messed up by the running of tests (so that subsequent tests can run in that environment. I don't see why I should have to package things into methods with magic names to have that happen. Instead, why not just have transactional instance variables? That way, I can use regular methods to set up the state for a test.
@order = Order.new("Dave Thomas", "Ruby Book") testing("normal case") do expect(@order.valid?) == true end testing("missing name in order") do @order.name = nil expect(@order.valid?) == false expect(@order.error) == "missing name" end # Check that order is reset to valid state here expect(@order.valid?) == trueSo, in the preceding case, the second
testingblock changed the@orderobject. However, once the block terminated, the object was restored to its initial (valid) state.





It looks great! Makes testing more natural and simple.
Posted by: Chillicoder | March 13, 2008 at 11:29 AM
So long as you don't release it until it's not a proof of concept anymore (*cough* RDoc *cough*), I think it looks awesome! ;)
Posted by: Jeremy | March 13, 2008 at 11:35 AM
Yes, please!
I like the grouping of rspec, but never got used to the `should`.syntax, the assert_equal felt more natural.
Your syntax looks great!
Posted by: Jonathan Weiss | March 13, 2008 at 11:39 AM
This looks so close to rspec. Plus it's got a fair mock api built in. And scenarios, though I haven't tried it yet, (http://faithfulcode.rubyforge.org/docs/scenarios/) might get you a per-test-group environment.
Posted by: Dan | March 13, 2008 at 12:08 PM
You should really check out the Haskell QuickCheck library.
Posted by: Aaron Denney | March 13, 2008 at 12:15 PM
Definitely yes. Curious if it involves ruby2ruby.
Posted by: Giles Bowkett | March 13, 2008 at 12:30 PM
Giles:
No external libraries used...
Dave
Posted by: Dave Thomas | March 13, 2008 at 12:36 PM
Totally, release it!
It would be interesting to see, how easy it will be to hack your library to suit existing tool chains, like Autotest, Rails and stuff like that.
Posted by: Hemant | March 13, 2008 at 01:45 PM
My only suggestion would be to allow nesting of "testing" groups.
Rspec recently added the ability to nest describe blocks, and its funny to say this, but I think it is by far the most important feature they've added in the last year that I've been using Rspec. Now I can group things together that are the same, and if there's another group that is nearly the same, but with one extra distinction, I can just nest it -- rather than duplicating the same setup code in the new group.
Posted by: Dan Kubb | March 13, 2008 at 01:50 PM
Dan:
Turns out that's already supported... :)
Dave
Posted by: Dave Thomas | March 13, 2008 at 01:54 PM
I love the way this looks. I'm skeptical about == always being the way I'd want to express an assertion, but not in such a way that I don't believe this will be cool. In general I can see myself using this as an improvement over test/unit without going into the syntax oddities that bother me in the BDD frameworks.
Posted by: Chad Fowler | March 13, 2008 at 03:57 PM
Jeremy, the phrase "perfect is the enemy of good" has become a cliche for a reason.
Posted by: Chad Fowler | March 13, 2008 at 04:05 PM
Am I the only person who doesn't think it "fairly natural"? Why would you write
expect(factorial(5)) == 120
instead of
expect(factorial(5) == 120)
?
Posted by: Alexey Romanov | March 13, 2008 at 04:40 PM
Please continue with this. It's lean, mean and clean.
Posted by: Jeremy Ashkenas | March 13, 2008 at 04:48 PM
Alexey:
If I did that, there'd be no way of changing the behavior of the comparison operators, so all I could do is report "success or failure". The way it is in the post, I have access to the values, too.
Dave
Posted by: Dave Thomas | March 13, 2008 at 05:01 PM
Looks very intriguing. I'd like to second Jeremy's suggestion to make sure it's mature before it's released, but it would be cool if you could give a preview of the code. :-)
Two things:
* "expect" is very much a testing mindset. Maybe something like "require" (except for the obvious problem in Ruby) would help guide people into thinking more in terms of specification?
* I often miss assertions for comparing two collections as sets and for simply picking out elements in a web page Rails' "assert_select" is a good start, but if this could take the same step from assert_equal with assert_select...
Posted by: Johannes Brodwall | March 13, 2008 at 05:09 PM
Johannes:
I'm not planning to work on this at all. I'll probably post some zygote code soon, and then anyone who wants to make something mature out of it will be free to do the nurturing.
Dave
Posted by: Dave Thomas | March 13, 2008 at 05:13 PM
Dave:
I see. Unlike Haskell, you can't introduce new operators for that purpose. What about
expect factorial(5), :==, 120
then?
Posted by: Alexey Romanov | March 13, 2008 at 05:32 PM
One of the things I really like about rSpec is the HTML report which documents all the specs and form a nice summary of things that are getting tested.
Perhaps this could be collected for a report from every testing() method.
testing("Customer should not allow destroy if it has any open orders.")
As an aside, I'm totally puzzled by the strict one-assertion-per-test advocates. When I write a test, I might have one main assertion as my goal, but I'm going to constrain things I think might go wrong along the way. I'm glad to see you grouping some expect()s there.
Posted by: Scott Schram | March 13, 2008 at 09:29 PM
Please do release this...and ignore the naysayers asking you to 'mature' it. I'd love to see this on a git repo somewhere and what kind of forks develop.
Posted by: Rob Sanheim | March 13, 2008 at 10:04 PM
Dave,
It may be a reasonable compromise for those using TestUnit but liking particular things from RSpec space.
Posted by: Michael Klishin | March 14, 2008 at 01:40 AM
Definitely, go ahead :)
Chad said:
> I'm skeptical about == always being the way I'd want to express an assertion
So am I, but I don't see why this won't allow to use >, <=, or even other methods. Custom matchers is probably the best win of rspec over t/u. Letting you take the tests into your app domain really does it for me :)
I'd expect the same from this library. Dave?
Posted by: Nicolás Sanguinetti | March 14, 2008 at 02:22 AM
Yes, it definitely it worth being developed into something mature.
Especially if it does not change the behaviour of Object, Kernel and/or other rather fundamental elements in any way (like RSpec does).
Posted by: Stephan | March 14, 2008 at 02:23 AM
That would be interesting to contrast this with another "expectations" library: http://blog.jayfields.com/2008/03/ruby-expectations-gem-version-023.html
At first glance, several testing operators are supported (like checking for an exception being thrown) and the expected value comes before the actual one, following the first xUnit libraries convention.
Eric.
Posted by: Eric Torreborre | March 14, 2008 at 02:54 AM
I like the syntax, too. But to be honest -- it is not Rocket Science. What I really like, is the way the tests are organized: no setup or teardown and documentation lives inside the comments. This is the great new thing here. It gives a new way of writing test code without all the boiler plate, that other libraries require.
Another solution for the expect syntax style could be copying from assert { 2.0 }. This way one could write something like:
expect{ factorial(5) == 120 }
which would be equally readable. But it will perhaps need external library support to analyze the contents of the block.
Posted by: Gregor Schmidt | March 14, 2008 at 04:13 AM