Understanding the power of isolated object testing

I spent some time last night watching J.B. Rainsberger's excellent InfoQ presentation entitled "Integration Tests are a Scam". I've been recently contemplating why I favor unit testing (or as J.B. calls it, isolated object testing) over integration testing and I found his presentation particularly relevant. I highly recommend watching it. He also has a series of blog entries that support the presentation (Parts 1, 2, and 3). I also recommend reading those. They're truly gems.

 

Testing to prove basic correctness

I find many developers using integrated tests as a way to prove the basic correctness of the class or system under test. J.B. writes that "While integration tests offer value in other contexts, too many programmers use them to show basic correctness, and when they do that they waste a tremendous amount of time and effort." Integrated testing can be used within a project (I'm personally fond of acceptance testing), but integrated testing should not be used to prove basic correctness of your code. Focused, isolated object tests (aka unit tests using test doubles) should be used for this endeavor. If you discover behavior that a collaborator demonstrates and you have not accounted for in your isolated object tests, you should mimic this behavior in your test doubled collaborator contracts. You want to cover as much of your code with isolated object tests. We'll talk more about contract tests later.

J.B. mentions using an integrated test to learn about how a collaborator might support its contract, but that this integrated test is not included in the basic correctness test suite. I'm wondering if there isn't some other test suite here that we could use to keep integrated tests that support our learning the runtime and external dependencies. This test suite would be run periodically, but is not part of the whole continuous integration process of building a software system. Need to noodle on this more.

 

Isolated object testing

J.B. states in his aforementioned presentation that he does not use the term "unit testing" and instead favors a more focused term of "isolated object testing". He makes a point to call out the isolated word; these tests isolate the class under test by using test doubles to stub or mock the collaborators of the class under test. These tests focus on a single object and a singular behavior. Any collaborations are realized using test doubles.

I tend to agree that the phrase "unit testing" is a weak phrase describing the type of testing that we use to drive design of a single class and a single method. These tests focus on the responsibility of a single method of a single class. These tests also help you focus on understanding the contracts of collaboration between this class under test and its collaborators. This is really where the design part of the equation comes in. Classes that are hard to test are screaming out that the design is wrong. The tests are saying you have too much responsibility in the class under test or there are too many collaborators. Using your fast isolated object tests, you can engage in a reactive design activity, moving responsibilities out of the class under test, create new collaborators, and other design changes to make .

 

The collaboration contract

J.B. rightfully makes a big deal about contracts (aka interfaces) to explicitly specify collaboration protocols between objects. He mentions that in Domain-Driven Design, popularized by the Eric Evans book of the same name, three concepts can be used to express a domain-driven model: Values, Entities, and Services. All Services should have contracts and those contracts manifest themselves as interfaces. By specifying interfaces, we explicitly declare the protocol supported by each interface implementation. Constraining ourselves to interface types when specifying collaborations results in looser coupled systems, which is considered a "Good Thing". When engaging in isolated object testing, J.B. details the concept of collaboration tests and the closely associated contract tests. This concept of collaboration tests and contract tests is something new to me and is a big reason the presentation was so valuable in my eyes.

 

Collaboration tests and contract tests

After watching the presentation, I seem to be doing a pretty good job of using collaboration tests, but I'm not making the association to contract tests. Collaboration tests prove that the client interacts with its collaborators correctly; the client sends the correct messages and message arguments to the collaborator and appropriately handles all outputs from the collaborator. This is traditionally what I have used mock objects for and that seems to be what J.B. is arguing one should do for collaboration testing.

Contract testing, on the other hand, deals with testing that an interface implementation accurately respects the interface it is implementing. Does the implementor support the contract it declares to support? I haven't typically written these types of tests, but I'm going to start. Interestingly, I don't see a lot of this sort of testing in the wild. The part I really dig about contract tests as Rainsberger explains them is that they can be reused across interface implementations. He uses the List interface and two of its implementations, ArrayList and LinkedList, and details how to use implementation inheritance in the tests to DRY up your contract tests.

It really gets interesting when he declares that for every collaboration test that implies that a test double behavior, there better be a corresponding contract test that demonstrates that the interface implementation(s) actually does support that behavior. Same goes for values returned: if a test double returns a value in a collaboration test, there should be a contract test that demonstrates that the real implementation(s) does actually return that value. This is where I think selective, integrated learning tests can help you discover how your classes on the edges of a system may act when integrated to real external dependencies. But again, those integrated tests are not providing basic code correctness semantics. They're in your project to help you learn, but are not part of the isolated object tests suite. Don't lump them with your isolated tests and don't run them as part of your code/update repos/run tests/commit cadence.  This learning test suite should be run periodically during the day, but not part of the CI build process.

The death spiral of integrated testing for proving basic code correctness

Both Rainsberger's tutorial presentation and his blog postings go into great detail to the fallacy of using integrated testing to prove code correctness. I won't rehash what he has to say about it. My conclusions that I draw from his material are:

• Focus on isolated object tests and isolation techniques and learn how to do collaboration and contract testing.
• If you need to learn, by all means, write an integrated test. If you need to reproduce a defect, initially write it as an integrated test. However, in all cases, take what you have learned and replicate that in your isolated object test suite.  Replicate the behavior you discovered in the integration test(s) with test doubles.  This will ensure that your isolated object tests stay true to the behavior of the integrated system.
• Don't get sucked into the downward death spiral of using integrated testing to guide your test-driven design efforts. It will only cause you pain.

J.B.'s series on Integrated Tests are a Scam can be found here.

 

Testing as a learning sandbox

I've been spending some quality time with NHibernate 3.0.  Last night I got stuck on an issue with the Criteria query where a collection passed to a constructor was null and the framework was complaining.  After writing a couple of integration tests that tested various parts of my domain object model, I was able to determine that collection types that I was using for the many-side of relationships (ISet<T> and HashSet<T> in this case) where the inappropriate collection types to be using for my collection semantics configuration.  I'm continually amazed at how powerful testing, both unit and integration, can be.  My tests today allowed me to create a sandbox to try things and work out a misunderstanding that I had with NHibernate.  Pretty cool.

Anatomy of a successful large agile project

I recently had a conversation with a colleague of mine at a company where I’m currently consulting.  We’ve been trying to bootstrap a collection of projects using an agile development process and associated software craftsmanship behaviors.   We have had mixed success to date.  Frustrated with the progress, my colleague asked me to enumerate what I felt were the success factors on the WestlawNext project that I had recently participated on.

I worked at Thomson Reuters from January 2008 to August 2010, during the initial releases of the WestlawNext project.  I was with the project from the beginning of its software development; the product development group had been working on the inception of the WestlawNext project for a couple of years prior.  WestlawNext was a very large project.  Hundreds of people were involved and millions of dollars were spent to build the next generation legal and regulatory research tool.  A lot was riding on this product.  It had to be a success—there was no option for failure.   The following themes are what I feel made this project a success.

 

Attitude

Now that I’ve had time to ponder my WestlawNext experience from afar, I think the number one reason for its success was attitude.  This was an audacious effort to build a new legal research tool in two years time with the number of people involved using an agile software development process.

But from the very beginning, a “can-do” attitude was instilled in the group that we would succeed.  We were going to “knock the ball out of the park” with this product.  There was never a thought that this thing might fail.  Many concerted efforts were made to continually propagate this attitude throughout the participants of the project.  Project tee-shirts, baseball trading cards, raffles, and summer socials were utilized to promote this team spirit.  This infectious attitude allowed us to overcome obstacles that would probably derail other projects.  People were willing to take responsibility for their work and put in the effort over and above the call of duty time and time again.

Communication

Communication is one of the most important functions of a software development project.  Large projects are very susceptible to communication breakdowns as the number of people increase.  We tried to minimize these breakdowns by favoring face-to-face communication as much as possible.  We were encouraged as software developers to pair program.  Designers were encouraged to work directly with developers on styling concerns.  We were encouraged to collaborate together when tough problems arose.  No GoToMeeting.  No conference calls.  Face-to-face conversations.

We were extremely lucky to be able to co-locate almost everyone on the project in three areas of the Thomson Reuters facility in Eagan.  When I mean everyone, I mean business people, vice presidents, directors, testers, designers, managers, coaches, and software developers.  This is one of a few places that I have consulted that have had the luxury of co-locating people in common areas.

 

Leadership

WestlawNext benefited from strong leadership that what 100% dedicated to the project.  No other obligations—they were focused solely on the development on the new product.  Our leaders were also quite familiar with agile software development process.  Some of them had come from other agile projects, both within the company and from outside.  They didn’t have to start from square one and many already knew the key behaviors of the process.  A few of them were software developers at one time (or still are). This is refreshing from a developer’s point of view.  They understand what it takes to build software; they’ve been in the trenches.

There is one moment in particular that I am very fond of.  I was working on a tough networking issue with some .NET code that we had provided another group.  We were throwing spurious socket closed exceptions, but it didn’t happen all the time and it seemed to occur only when the server load increased.  Our senior director, one of our leaders, was helping triage the issue and participating in our root cause analysis.  This senior director had technical chops and was quite proficient at network analysis.  He rolled up his sleeves and got right in and loved being able to help solve the issue.  We did solve the issue; it turned out to be tied to a deprecated thread-local storage API in .NET.  That leader earned a ton of my respect that day.

 

Testing

Testing is paramount to building a quality software product.  The WestlawNext project embraced testing like I have never seen before in my career.  We evolved our designs with unit testing.  We used integration testing to ensure that software components were wired together correctly.  Acceptance testing ensured that features did not regress in future iterations of development.  Load and performance testing was continuously run in an effort to tune the overall product.  Beta testers were allowed to play with the software well in advance of its initial release date, ensuring that it satisfied the customer.

All of this testing allowed us to build tight feedback loops, giving us near-instantaneous data on the health of our growing and evolving product.  The suites of tests infused confidence within the project group; we knew exactly how the software performed at specific load levels.  I cannot fathom working on a software development project that does not fully embrace the aforementioned levels of testing.

 

Conclusion

In conclusion, I’m starting to realize that the WestlawNext project may have been one of those rare moments where everything came together in near perfect harmony to produce a great product.  As I have moved on from Thomson Reuters, I yearn to replicate a similar experience at my other clients.  My current engagement only reinforces the fact that every software development project takes a different path to success, and some may never make it to the end.

 

 

 

Autofixture: A generic Test Data Builder implementation for .NET

Just came across a Test Data Builder implementation for .NET, Autofixture.  The Test Data Builder pattern has become quite popular recently since it was mentioned in Growing Object-Oriented Software, Guided by Tests.  I've used the pattern before, but I've always built the builder implementations by hand.  This implementation looks really promising.

Test-Driven in Groovy presentation in the bag

Joe Muraski and myself did a Test-Driven in Groovy workshop at GR8 in the US conference here in Bloomington, MN this past Friday. If you're interested in the materials (presentation, demo app), you can get it all right here. The demo consists of a Java web app with lots of opportunity for refactoring, Groovy unit tests, and Cucumber (via cuke4duke) acceptance tests.

Mockito's @InjectMocks annotation does reflection-based DI

I've been using @InjectMocks heavily lately since it came out in version 1.8.3 of mockito. The javadocs for this annotation state that it uses setter injection to inject its dependencies. Being a lazy developer, I was writing a unit test yesterday at work and forgot to write the setter methods for a couple of dependencies on the SUT that I was testing. Lo and behold, the test passed and all mock verifications were satisfied. Very confused, I went back to some other unit tests and their SUTs and removed the setter methods. In all of my cases, the tests continued to pass. A quick note to the mockito list confirms what I discovered--@InjectMocks actually is using a reflection-based DI scheme, not unlike what Spring does when you annotate collaborator fields in a Spring bean with @Autowired and @Resource. Very cool feature, as this further reduces code noise. Here is the discussion on Google Groups.

New annotations in mockito 1.8.3

Mockito 1.8.3 was recently released and I got a chance to use it today. There are a couple of new annotations available in this release: @Captor, @Spy, and @InjectMocks. I was able to use @Captor and @InjectMocks today. Even after just a few minutes with these two annotations, I'm sold. Very cool enhancement. I spent some quality time with EasyMock last week, and there's nothing easy with EasyMock. Ugh!! If you're hip to using mock objects in your Java unit testing efforts, you really should look at mockito these days. Details about @Captor and @InjectMocks follows.

@Captor

This annotation will automatically create typed argument captors (org.mockito.ArgumentCaptor<T>) in your unit tests. Argument captors are essential in verifying indirect outputs to your mocked collaborators.

public class Test{
    @Captor ArgumentCaptor<Foobar> foobarCaptor;

    @Before
    public void init(){
       MockitoAnnotations.init(this);
    }

    @Test 
    public void shouldDoSomethingUseful() {
       //...
       verify(mock.doStuff(foorbarCaptor.capture()));
       Foobar capturedFoobar = foobarCaptor.getValue();
       assertEquals("foobar", capturedFoobar.getName());
    }
 }

@InjectMocks

Automatically injects mocks by type using setter injection. Constructor injection is not currently available, but if you want to provide a patch, the mockito team will gladly consider your contribution. I'm actually more interested in reflection-based injection, similar to what Spring uses when annotating dependency fields using @Autowired. Having your unit tests inject dependencies via reflection would help me avoid the set* methods on the implementations. I may have to play with this a bit.

public class FooBarManagerTests  {

       @Mock private FooDependency mockFoo;
       @Mock private BarDependency mockBar;
       @InjectMocks private FooBarManager manager = new FooBarManagerImpl();

       @Before 
       public void initMocks() {
           // Initializes all mocks and then injects those mocks into the FooManager instance.
           MockitoAnnotations.initMocks(this);
       }

       @Test 
       public void shouldDoSomething() {
           manager.doSomething();
           verify(mockFoo).doSomethingToFoo(any(String.class));
           verify(mockBar).doSomethingToBar(any(Integer.class));
       }
   }

Using argument matchers in EasyMock and mockito

I spent some quality time with EasyMock today while coaching some developers on building tests around an existing codebase. We were mocking a dependency of our SUT and the method call that we wanted to mock had two parameters. When we wrote the expectation, we used the anyObject() matcher for the first parameter and tried to perform an exact match on the second parameter, a Boolean. Here is an example of what we were trying to do in EasyMock (NOTE: we statically imported EasyMock so we could reduce the amount of code noise by not prefacing our expect, matcher, replay and verify invocations with EasyMock.):


expect(mockObject.retrieveSomething((String) anyObject(), false)).andReturn(someObject);


Unfortunately, EasyMock and mockito do not like this. They both want you to use matchers for all parameters if you use matchers for any parameters. However, the two libraries react quite differently when this situation occurs. EasyMock complains with a somewhat confusing message that at first blush makes it seem like we declared the expectation for multiple invocations. It really threw us off for a while (at least an hour) trying to figure out what was wrong with our expectations. Here is how we fixed it in EasyMock:


expect(mockObject.retrieveSomething((String) anyObject(), eq(false))).andReturn(someObject);


Mockito does a much better job of stating that when you use a parameter argument matcher in your expectation, you have to use parameter argument matchers for all of the parameters of the method call participating in the expectation. I find it interesting that mockito retains the behavior of EasyMock (mockito is a fork of EasyMock) with regards to argument matching, but mockito improves on the error messaging when something goes wrong with the mock object setup. Further reinforces my decision to forego EasyMock in favor of mockito.