Playing with a Testing Library

• I'd like to be able to express my unit tests fairly naturally, using the conditional operators built into the language. So, for example, I'd want to write:

  
        expect(factorial(5)) == 120
        expect(factorial(10)) > 10000
      

• I'd like the error messages to show both the code that caused the error and the values that caused the error. So, for example, I'd want the following (incorrect) test

  
        expect(factorial(6)) == 600
      

  to output something like

        /Users/dave/tmp/tmc/blog_tests.rb:16
            the code was: expect(factorial(6)) == 600,
            but 720 != 600
      

  and

  
        expect(1) > 2
      

  should say

  
        /Users/dave/Play/tmc/blog_tests.rb:11
        	the code was: expect(1) > 2,
        	but 1 <= 2
      

  (Note how the expression showing the actual values negates the comparison operator to make it easier to read.)

• I annotate my code with comments, so I'd like to be able to annotate my tests the same way.

  
      expect(factorial(6)) == 600  # Deliberate bad test
    

  should produce something like

      /Users/dave/tmp/tmc/blog_tests.rb:17
         Deliberate bad test
         the code was: expect(factorial(6)) == 600,
         but 720 != 600  
    

  Sometimes I write longer comments.

  
     # The factorial of 6 is a special case,
     # because of the labor laws in Las Vegas
     expect(factorial(6)) == 600
     

  So the resulting errors are longer, too.

             
     /Users/dave/Play/tmc/blog_tests.rb:21
        The factorial of 6 is a special case, because of the labor laws in Las Vegas
        the code was: expect(factorial(6)) == 600,
        but 720 != 600
     

• I like to be able to group my tests.

        testing("positive factorials") do
          expect(factorial(1)) == 1
          expect(factorial(2)) == 2
          expect(factorial(5)) == 120
        end
  
        testing("factorial of zero") do
          expect(factorial(0)) == 1
        end
  
        testing("negative factorials") do
          expect(factorial(-1)) == 1
          expect(factorial(-5)) == 1
        end
      

• I like the description of the group to appear along with any individual test annotation if a test fails.

  
        testing("factorial of zero") do
          # this test is deliberately wrong
          expect(factorial(0)) == 0
        end
      

  will produce

        /Users/dave/Play/tmc/blog_tests.rb:31--while testing factorial of zero
            this test is deliberately wrong
            the code was: expect(factorial(0)) == 0,
            but 1 != 0
      

• I like to have the flexibility to set up the environment for a group of tests. I also like to have the idea of a global environment which doesn't get messed up by the running of tests (so that subsequent tests can run in that environment. I don't see why I should have to package things into methods with magic names to have that happen. Instead, why not just have transactional instance variables? That way, I can use regular methods to set up the state for a test.

        @order = Order.new("Dave Thomas", "Ruby Book")
  
        testing("normal case") do
          expect(@order.valid?) == true
        end
  
        testing("missing name in order") do
          @order.name = nil
          expect(@order.valid?) == false
          expect(@order.error)  == "missing name"
        end
  
        # Check that order is reset to valid state here
        expect(@order.valid?) == true
      

  So, in the preceding case, the second testing block changed the @order object. However, once the block terminated, the object was restored to its initial (valid) state.

So, while waiting for the last day of the Rails Studio to start, I hacked together a quick proof of concept. It's less than 100 lines of code. All the output shown here was generated by it. Is this worth developing into something usable?

Comments

[Chillicoder]

It looks great! Makes testing more natural and simple.

[Jeremy]

So long as you don't release it until it's not a proof of concept anymore (*cough* RDoc *cough*), I think it looks awesome! ;)

[Jonathan Weiss]

Yes, please!

I like the grouping of rspec, but never got used to the `should`.syntax, the assert_equal felt more natural.

Your syntax looks great!

[Dan]

This looks so close to rspec. Plus it's got a fair mock api built in. And scenarios, though I haven't tried it yet, (http://faithfulcode.rubyforge.org/docs/scenarios/) might get you a per-test-group environment.

[Aaron Denney]

You should really check out the Haskell QuickCheck library.

[Giles Bowkett]

Definitely yes. Curious if it involves ruby2ruby.

[Dave Thomas]

Giles:

No external libraries used...

Dave

[Hemant]

Totally, release it!

It would be interesting to see, how easy it will be to hack your library to suit existing tool chains, like Autotest, Rails and stuff like that.

[Dan Kubb]

My only suggestion would be to allow nesting of "testing" groups.

Rspec recently added the ability to nest describe blocks, and its funny to say this, but I think it is by far the most important feature they've added in the last year that I've been using Rspec. Now I can group things together that are the same, and if there's another group that is nearly the same, but with one extra distinction, I can just nest it -- rather than duplicating the same setup code in the new group.

[Dave Thomas]

Dan:

Turns out that's already supported... :)

Dave

[Chad Fowler]

I love the way this looks. I'm skeptical about == always being the way I'd want to express an assertion, but not in such a way that I don't believe this will be cool. In general I can see myself using this as an improvement over test/unit without going into the syntax oddities that bother me in the BDD frameworks.

[Chad Fowler]

Jeremy, the phrase "perfect is the enemy of good" has become a cliche for a reason.

[Alexey Romanov]

Am I the only person who doesn't think it "fairly natural"? Why would you write

expect(factorial(5)) == 120

instead of

expect(factorial(5) == 120)

?

[Jeremy Ashkenas]

Please continue with this. It's lean, mean and clean.

[Dave Thomas]

Alexey:

If I did that, there'd be no way of changing the behavior of the comparison operators, so all I could do is report "success or failure". The way it is in the post, I have access to the values, too.

Dave

[Johannes Brodwall]

Looks very intriguing. I'd like to second Jeremy's suggestion to make sure it's mature before it's released, but it would be cool if you could give a preview of the code. :-)

Two things:
* "expect" is very much a testing mindset. Maybe something like "require" (except for the obvious problem in Ruby) would help guide people into thinking more in terms of specification?

* I often miss assertions for comparing two collections as sets and for simply picking out elements in a web page Rails' "assert_select" is a good start, but if this could take the same step from assert_equal with assert_select...

[Dave Thomas]

Johannes:

I'm not planning to work on this at all. I'll probably post some zygote code soon, and then anyone who wants to make something mature out of it will be free to do the nurturing.

Dave

[Alexey Romanov]

Dave:
I see. Unlike Haskell, you can't introduce new operators for that purpose. What about

expect factorial(5), :==, 120

then?

[Scott Schram]

One of the things I really like about rSpec is the HTML report which documents all the specs and form a nice summary of things that are getting tested.

Perhaps this could be collected for a report from every testing() method.

testing("Customer should not allow destroy if it has any open orders.")

As an aside, I'm totally puzzled by the strict one-assertion-per-test advocates. When I write a test, I might have one main assertion as my goal, but I'm going to constrain things I think might go wrong along the way. I'm glad to see you grouping some expect()s there.

[Rob Sanheim]

Please do release this...and ignore the naysayers asking you to 'mature' it. I'd love to see this on a git repo somewhere and what kind of forks develop.

[Michael Klishin]

Dave,

It may be a reasonable compromise for those using TestUnit but liking particular things from RSpec space.

[Nicolás Sanguinetti]

Definitely, go ahead :)

Chad said:
> I'm skeptical about == always being the way I'd want to express an assertion

So am I, but I don't see why this won't allow to use >, <=, or even other methods. Custom matchers is probably the best win of rspec over t/u. Letting you take the tests into your app domain really does it for me :)

I'd expect the same from this library. Dave?

[Stephan]

Yes, it definitely it worth being developed into something mature.
Especially if it does not change the behaviour of Object, Kernel and/or other rather fundamental elements in any way (like RSpec does).

[Eric Torreborre]

That would be interesting to contrast this with another "expectations" library: http://blog.jayfields.com/2008/03/ruby-expectations-gem-version-023.html

At first glance, several testing operators are supported (like checking for an exception being thrown) and the expected value comes before the actual one, following the first xUnit libraries convention.

Eric.

[Gregor Schmidt]

I like the syntax, too. But to be honest -- it is not Rocket Science. What I really like, is the way the tests are organized: no setup or teardown and documentation lives inside the comments. This is the great new thing here. It gives a new way of writing test code without all the boiler plate, that other libraries require.

Another solution for the expect syntax style could be copying from assert { 2.0 }. This way one could write something like:

expect{ factorial(5) == 120 }

which would be equally readable. But it will perhaps need external library support to analyze the contents of the block.