Understanding the power of isolated object testing

I spent some time last night watching J.B. Rainsberger's excellent InfoQ presentation entitled "Integration Tests are a Scam". I've been recently contemplating why I favor unit testing (or as J.B. calls it, isolated object testing) over integration testing and I found his presentation particularly relevant. I highly recommend watching it. He also has a series of blog entries that support the presentation (Parts 1, 2, and 3). I also recommend reading those. They're truly gems.

 

Testing to prove basic correctness

I find many developers using integrated tests as a way to prove the basic correctness of the class or system under test. J.B. writes that "While integration tests offer value in other contexts, too many programmers use them to show basic correctness, and when they do that they waste a tremendous amount of time and effort." Integrated testing can be used within a project (I'm personally fond of acceptance testing), but integrated testing should not be used to prove basic correctness of your code. Focused, isolated object tests (aka unit tests using test doubles) should be used for this endeavor. If you discover behavior that a collaborator demonstrates and you have not accounted for in your isolated object tests, you should mimic this behavior in your test doubled collaborator contracts. You want to cover as much of your code with isolated object tests. We'll talk more about contract tests later.

J.B. mentions using an integrated test to learn about how a collaborator might support its contract, but that this integrated test is not included in the basic correctness test suite. I'm wondering if there isn't some other test suite here that we could use to keep integrated tests that support our learning the runtime and external dependencies. This test suite would be run periodically, but is not part of the whole continuous integration process of building a software system. Need to noodle on this more.

 

Isolated object testing

J.B. states in his aforementioned presentation that he does not use the term "unit testing" and instead favors a more focused term of "isolated object testing". He makes a point to call out the isolated word; these tests isolate the class under test by using test doubles to stub or mock the collaborators of the class under test. These tests focus on a single object and a singular behavior. Any collaborations are realized using test doubles.

I tend to agree that the phrase "unit testing" is a weak phrase describing the type of testing that we use to drive design of a single class and a single method. These tests focus on the responsibility of a single method of a single class. These tests also help you focus on understanding the contracts of collaboration between this class under test and its collaborators. This is really where the design part of the equation comes in. Classes that are hard to test are screaming out that the design is wrong. The tests are saying you have too much responsibility in the class under test or there are too many collaborators. Using your fast isolated object tests, you can engage in a reactive design activity, moving responsibilities out of the class under test, create new collaborators, and other design changes to make .

 

The collaboration contract

J.B. rightfully makes a big deal about contracts (aka interfaces) to explicitly specify collaboration protocols between objects. He mentions that in Domain-Driven Design, popularized by the Eric Evans book of the same name, three concepts can be used to express a domain-driven model: Values, Entities, and Services. All Services should have contracts and those contracts manifest themselves as interfaces. By specifying interfaces, we explicitly declare the protocol supported by each interface implementation. Constraining ourselves to interface types when specifying collaborations results in looser coupled systems, which is considered a "Good Thing". When engaging in isolated object testing, J.B. details the concept of collaboration tests and the closely associated contract tests. This concept of collaboration tests and contract tests is something new to me and is a big reason the presentation was so valuable in my eyes.

 

Collaboration tests and contract tests

After watching the presentation, I seem to be doing a pretty good job of using collaboration tests, but I'm not making the association to contract tests. Collaboration tests prove that the client interacts with its collaborators correctly; the client sends the correct messages and message arguments to the collaborator and appropriately handles all outputs from the collaborator. This is traditionally what I have used mock objects for and that seems to be what J.B. is arguing one should do for collaboration testing.

Contract testing, on the other hand, deals with testing that an interface implementation accurately respects the interface it is implementing. Does the implementor support the contract it declares to support? I haven't typically written these types of tests, but I'm going to start. Interestingly, I don't see a lot of this sort of testing in the wild. The part I really dig about contract tests as Rainsberger explains them is that they can be reused across interface implementations. He uses the List interface and two of its implementations, ArrayList and LinkedList, and details how to use implementation inheritance in the tests to DRY up your contract tests.

It really gets interesting when he declares that for every collaboration test that implies that a test double behavior, there better be a corresponding contract test that demonstrates that the interface implementation(s) actually does support that behavior. Same goes for values returned: if a test double returns a value in a collaboration test, there should be a contract test that demonstrates that the real implementation(s) does actually return that value. This is where I think selective, integrated learning tests can help you discover how your classes on the edges of a system may act when integrated to real external dependencies. But again, those integrated tests are not providing basic code correctness semantics. They're in your project to help you learn, but are not part of the isolated object tests suite. Don't lump them with your isolated tests and don't run them as part of your code/update repos/run tests/commit cadence.  This learning test suite should be run periodically during the day, but not part of the CI build process.

The death spiral of integrated testing for proving basic code correctness

Both Rainsberger's tutorial presentation and his blog postings go into great detail to the fallacy of using integrated testing to prove code correctness. I won't rehash what he has to say about it. My conclusions that I draw from his material are:

• Focus on isolated object tests and isolation techniques and learn how to do collaboration and contract testing.
• If you need to learn, by all means, write an integrated test. If you need to reproduce a defect, initially write it as an integrated test. However, in all cases, take what you have learned and replicate that in your isolated object test suite.  Replicate the behavior you discovered in the integration test(s) with test doubles.  This will ensure that your isolated object tests stay true to the behavior of the integrated system.
• Don't get sucked into the downward death spiral of using integrated testing to guide your test-driven design efforts. It will only cause you pain.

J.B.'s series on Integrated Tests are a Scam can be found here.

 

Autofixture: A generic Test Data Builder implementation for .NET

Just came across a Test Data Builder implementation for .NET, Autofixture.  The Test Data Builder pattern has become quite popular recently since it was mentioned in Growing Object-Oriented Software, Guided by Tests.  I've used the pattern before, but I've always built the builder implementations by hand.  This implementation looks really promising.

Test-Driven in Groovy presentation in the bag

Joe Muraski and myself did a Test-Driven in Groovy workshop at GR8 in the US conference here in Bloomington, MN this past Friday. If you're interested in the materials (presentation, demo app), you can get it all right here. The demo consists of a Java web app with lots of opportunity for refactoring, Groovy unit tests, and Cucumber (via cuke4duke) acceptance tests.

New annotations in mockito 1.8.3

Mockito 1.8.3 was recently released and I got a chance to use it today. There are a couple of new annotations available in this release: @Captor, @Spy, and @InjectMocks. I was able to use @Captor and @InjectMocks today. Even after just a few minutes with these two annotations, I'm sold. Very cool enhancement. I spent some quality time with EasyMock last week, and there's nothing easy with EasyMock. Ugh!! If you're hip to using mock objects in your Java unit testing efforts, you really should look at mockito these days. Details about @Captor and @InjectMocks follows.

@Captor

This annotation will automatically create typed argument captors (org.mockito.ArgumentCaptor<T>) in your unit tests. Argument captors are essential in verifying indirect outputs to your mocked collaborators.

public class Test{
    @Captor ArgumentCaptor<Foobar> foobarCaptor;

    @Before
    public void init(){
       MockitoAnnotations.init(this);
    }

    @Test 
    public void shouldDoSomethingUseful() {
       //...
       verify(mock.doStuff(foorbarCaptor.capture()));
       Foobar capturedFoobar = foobarCaptor.getValue();
       assertEquals("foobar", capturedFoobar.getName());
    }
 }

@InjectMocks

Automatically injects mocks by type using setter injection. Constructor injection is not currently available, but if you want to provide a patch, the mockito team will gladly consider your contribution. I'm actually more interested in reflection-based injection, similar to what Spring uses when annotating dependency fields using @Autowired. Having your unit tests inject dependencies via reflection would help me avoid the set* methods on the implementations. I may have to play with this a bit.

public class FooBarManagerTests  {

       @Mock private FooDependency mockFoo;
       @Mock private BarDependency mockBar;
       @InjectMocks private FooBarManager manager = new FooBarManagerImpl();

       @Before 
       public void initMocks() {
           // Initializes all mocks and then injects those mocks into the FooManager instance.
           MockitoAnnotations.initMocks(this);
       }

       @Test 
       public void shouldDoSomething() {
           manager.doSomething();
           verify(mockFoo).doSomethingToFoo(any(String.class));
           verify(mockBar).doSomethingToBar(any(Integer.class));
       }
   }

Completed another Test Driven and Refactoring course for DevJam

This time using C# and the .NET platform. The course went over really well, though we did have some small snafus with the training area in the DevJam office. Nothing that can't be tweaked. I had 15 participants in this class. All participants pair programmed when completing the hand-on exercises and again, I'm totally amazed at how well pair programming goes over when you get people away from their normal work environment. Everyone really grooved on the pair programming thing.

One area that we will need to work on is the mock objects content. We don't have any hands-on exercises for using mock objects and we heard about it in the reviews of the course. I did walk everyone through a demonstration of using mock objects in your unit tests, but I mis-gauged how much interest the participants had in mock objects and the desire to get their feet wet with mock objects. Some of the class participants stay after the course ended and we did another 40 minutes of live coding demos on the use and features of mock objects (using moq for the mocking framework in .NET).

All in all, an awesome two days for me and hopefully for the course participants.

Know your SUT and your mocks

I was working with a colleague of mine tonight and he became confused with his unit test. It was an interesting exchange, so I thought I would write it up. He had a system under test (SUT), a service, that had a number of dependencies. The dependencies were mocked using a mocking framework (mockito in this instance). He was writing a test to get better code coverage in preparation for some refactoring and he started to add an expectation to the SUT. When he ran his test, the test failed, but the error lead you to believe that it was the improper use of matchers within mockito. It took him and I awhile to realize that he was trying to add expectations to a non-mock class. In hindsight, he should have picked up on this right away, as we have a coding convention of naming our mock object instances as mock*. The SUT does not follow this naming convention and hence you should see the problem right away. A little concerned that mockito didn't alert us to the fact that we were trying to add expectations to a non-mock instance. Moral of the story: Make your mock object instances stand out--name them appropriately so everyone knows that it's a mock.

Using ArgumentMatcher to capture indirect output arguments

Yesterday, I posted a blog entry about performing assertions directly in your ArgumentMatcher implementations.  I wanted to close the loop on my previous usage of the ArgumentMatcher.  After some discussion with the main committer to mockito, Szczepan Faber, it seems that ArgumentMatcher should only be used to capture the indirect output argument, making it accessible to the test code. Once the test code has the indirect output argument available to it, you can then assert on it to your heart's delight. Szczepan recommended not embedding the assertions directly in the ArgumentMatcher implementation.  More information can be found here.  Seems like a good approach. 

.NET Base Class Library types not friendly to mocking with MoQ

I spent some quality time this week driving a design implemented in .NET using unit tests.  It was an enlightening experience and gave me a good glimpse at the quality of the BCL design.  I'm not impressed. We had issues trying to mock System.Net.WebRequest and System.Net.WebResponse and its decendents.  It seems that Microsoft prefers to expose abstract base classes, instead of using interfaces for this.  The aforementioned types are not interface types; they're abstract classes.  We've been using MoQ, a fine mocking framework, at the client I've been working at and I tried to create mock implementations of the System.Net.WebRequest and System.Net.WebResponse types.  No go--these types do not have public constructors.  The non-virtual, non-abstract methods also give MoQ headaches (good write up here about why methods of these abstract class types have to be explicitly decorated with virtual if you want to mock them).  Personally, I'm more fond of exposing interface types, thus alleviating the problem all together.  Very frustrating.  We did continue to unit test and utilized mocks for this, but it forced us to wrap these BCL types with our own types where we could control the virtual decoration.  Yuck!!