Shoulda used this earlier

In many ways, testing software is like going out and getting exercise. You know you should do it, and you know it does you good, but it's also pretty easy to find an excuse to skip it (I'll make it up tomorrow).

So anything that makes testing easier is good, because it cuts down on the excuses not to do it.

One thing I've never really liked about the conventional xUnit-style testing frameworks was the setup and teardown structure. In these frameworks, a test case is a class, and setup and teardown are implemented by methods in that class. Each test is also a method, so the basic flow is

  for each test method in the class
    run setup
    run the test method
    run teardown
  end

Nice and simple. Each test method got the benefit of a standard environment created by the setup method, and the teardown method got the job of tidying up after.

Except… when I'm writing tests, I typically want to set up lots of different scenarios. I'll want A and B and C, then A and B but not C, then A and not B, then A and D, and so on. I had two choices—write lots of test case classes, using subclassing to inherit common setup behavior, or write per-test method setup code (often factored out into helpers). In the end, I almost always did the latter, And that was tedious, and it made it harder to see the tests for the setup code.

I flirted with RSpec. Its spec framework seemed to have what I wanted. But I just couldn't get myself to enjoy using it. (I think it's a cat people/dog people kind of thing)

Enter shoulda

Then, a couple of weeks back, Mike Clark and Chad Fowler introduced me to shoulda. Shoulda isn't a testing framework. Instead, it extends Ruby's existing Test::Unit framework with the idea of test contexts. A context is a section of your test case where all the test methods have something in common. At it simplest, a context could be simply used as an annotation device (and, yes, this is a silly example):

context "My factorial method" do
  should "return 1 when passed 0" do
    assert_equal 1, fact(0)
  end
  should "return 1 when passed 1" do
    assert_equal 1, fact(1)
  end
  should "return 6 when passed 3" do
    assert_equal 6, fact(3)
  end
end    

The stuff in a context can share common setup code—just write a setup block.

class CartTest < Test::Unit::TestCase

  context "An empty cart" do
    setup do
      @cart = orders(:wilmas_empty_cart)
    end

    should "have no line items" do
      assert_equal 0, @cart.line_items.size
    end

    should "have a zero price" do
      assert_equal 0, @cart.price
    end
  end

  context "Some other context..." ...
  end
end

So now, within a single test case I can set up multiple contexts, and each context can have its own environment.

But, take it back to my original problem. I often want to set up hierarchies of related environments for my tests. The shaoulda code handles this wonderfully, because it lets me nest contexts. For example, I'm adding a feature to our store that gives customers some additional information if, during checkout, their credit card transaction was initially rejected because the address was wrong, and was then accepted when they fixed the address. I wanted two tests, one without the prior address error, and one with.

To set up this environment, I needed to set up a shopping cart, create a dummy response from our payment gateway, and post that response to the application. In the case of the prior address error, I also wanted to inject an entry containing that error into the transactions associated with the order prior to generating the response.

With shoulda, I simply created some nested contexts. The top level context did the shared setup, and the inner contexts then set up appropriate environments for their tests. It looked like this:

  context "Checking out"  do
    setup do
      @cart = cart_named(:freds_full_cart)
      @cart.prepare_for_store_authorize!
      @params = approved_authnet_response(@cart)
    end                  
    
    context "with no AVS errors in CC transaction history" do
      setup do
        post :post_from_authnet_authorize, @params
      end

      should_redirect_to "{:action => :receipt}"
    end 
    
    context "with AVS errors in CC transaction history" do
      setup do
        avs_error = CcTransaction.new(:response_code => 2, :response_reason_code => 27)
        @cart.cc_transactions << avs_error
        post :post_from_authnet_authorize, @params
      end

      should_redirect_to "{:action => :explain_avs_mismatch}"
    end
  end 

The outer setup gets run before the execution of each of the inner contexts. And the setup in the inner contexts gets run when running that context. And shoulda keeps track of it all, so I get very natural error messages if an assertion fails. For example, if the test in the second context above fails, I'd get

Checking out with AVS errors in CC transaction history should 
redirect to "{:action => :explain_avs_mixsmatch}". 

So, now, I can finally set up my hierarchies of test environments in a natural way. It isn't revolutionary. It's just one less excuse for not testing…

Comments

[Sam Aaron]

This looks remarkably similar to the structure of rspec tests: just switch context for describe, should for it, setup for before(:each) and you're pretty much there. However, this doesn't introduce the marmite-like 'love it or hate it' syntactic sugar/vinegar that people rant about.

I think that nesting contexts like this is a great idea, and it's good to see it over on the Test::Unit side of the fence :-)

[Phil]

@Sam: actually it looks more like older versions of RSpec; they used context liberally, and I think setup as well. But I'm not sure if nesting was supported.

[Morgan Roderick]

I've been using shoulda for awhile ... I guess I should have told you about it ;-)

Nice to see that it's getting some traction, as it's very nice, and keeps your testing nice and easy to manage.

RSpec just seemed to big me when I was flirting with it, but with shoulda, the glove just fits me perfectly.

[Dan Croak]

I'm a dog person.

[Ben]

RSpec actually explicitly avoided nested contexts for quite some time - they're a more recent addition.

[Sebastian]

Very usefull post.

[Husein Choroomi]

What I love about Shoulda is its great macros [1] and how easy it's to add new macros.

[1]
http://dev.thoughtbot.com/shoulda/classes/ThoughtBot/Shoulda/Controller/ClassMethods.html
http://dev.thoughtbot.com/shoulda/classes/ThoughtBot/Shoulda/ActiveRecord.html

[Adrian Kuhn]

I wonder if what you do could also be achieved with JExample's first-class dependencies [1]. JExample runs tests based on a dependency graph, and you can run a subset/subgraph of the test. Would be a interesting exercise to port JExample to Ruby anyway ... there is certainly a nice Rubyism that can be used instead of Java's annotations.

[1] http://www.iam.unibe.ch/~scg/Research/JExample/

[Robert Walker]

I haven't yet landed on a testing framework for my Ruby work. I'm still pretty new to Ruby and don't have anything currently in production. I've looked at Test::Unit, but I've been investigating rSpec in much more depth. I have heard of Shoulda mentioned in passing, but I've not looked at it yet.

Before I dig too deeply, I was wondering about what people are using for mocking and stubbing with Shoulda. Apart from the nice syntactic sugar provided by rSpec, its mocking and stubbing features seem pretty powerful and very useful. I am aware of some mocking frameworks like Mocha, but is there anything built into Shoulda? If not then how easy is it to integrate Shoulda with a mocking/stubbing framework?

[Chad Pytel]

Robert, we use Mocha with Shoulda, and its just a matter of installing them both.

[Tammer Saleh]

Adrian: JExample looks pretty interesting. Basically, it combines the ideas of a test and a setup into one unit, which contains assertions and will only run the units that depend on it once those assertions pass. This feels to me like a superset of the features of Shoulda, so I'm inclined to say that there's technically nothing Shoulda can do that JExample can't.

That being said, one of the main goals with Shoulda was to create a full featured testing system built on top of the ruby Test::Unit library. This allows easy and consistent migration for existing projects. I don't think it would be possible or practical to do that with JExample.

It would, however, be very interesting to see a new ruby testing framework that follows this dependency-graph pattern.

[Chad Humphries]

Yep, I would agree with the earlier commenters, this has the feel of rspec from about a year ago. Nothing wrong with that though :) There macros are pretty cool, but not really any different than any of the RSpec rails matcher plugins out there.

[edovale]

Just a counter comment about the xUnit comment.
This is like running parametrized tests. JUnit 4 has this ability where you can pass parameter to the test method and for every provided combination the framework will give the test method a run.

[Chris]

I wrote tests that looked almost exactly like this a year or so ago...in RSpec. I'm not sure if Shoulda has anything like it_should_behave_like, but I'd be interested to hear a comparison of the two frameworks from someone who knows what they're talking about. I've been using RSpec for a couple years now and love it, but I've never really explored Shoulda. And yup, the Shoulda macros just look like RSpec matchers, which you can write on your own or use plugins written by others.