When is a user story done?

I am beginning to hate the word "done". It means so many different things in software development and causes so much confusion that I am trying not to use the term, instead opting for statements which convey much more information and provide more transparency into our product development efforts.

In the past, I have been asked to figure out why two development groups under a common work area have radically different velocities and thus are being viewed much differently by management. The first group is chewing up stories, the other, not so much. Looking at the user stories of each group, differences between the two groups definitely exist. The "higher performing" group have simple stories with a few acceptance criteria. The acceptance criteria seem very high level; my gut feeling is that the acceptance criteria would be difficult to turn into actual acceptance tests. The "lower performing" group has more acceptance criteria which are more concrete--I can visualize actual acceptance tests with these criteria. Digging into the groups a bit more, it becomes apparent that the "higher performing" group's quality assurance effort is behind the development group in completion, but the group is considering stories "done" when the developers reach code complete. The "lower performing" group does not claim a story completed until it has been verified by quality assurance and the customer has signed off on the functionality (usually through some demonstration of the functionality).

So which group is really the high performer here? Saying a story is done when code complete is a misnomer and is likely building a false reality of where the group really is at. This "higher performing" group is setting themselves up for a big fall--there is no sense of quality assurance complete or customer complete.

Some in the agile community call this "done, done, done". I guess that's OK, but I think we need to be really careful with our terminology as we communicate within our development groups and outside of the development group to shareholders. I feel we should use specific terminology to describe where a story currently resides in the "done" spectrum. Daniel Gullo has a good article about this. He enumerates criteria for completion:

• “Code complete”: Development completed, including accompanying unit tests (assumption here is that everyone is operating under test-driven development).
• “Unit tested”: Unit tests completed. I don't really care for this designation, especially since my groups engage in test-driven development. I would call this Acceptance tested or something like that. Basically, demonstrate that we have satisfied the conditions for completion (aka acceptance criteria). Hopefully most of these acceptance tests are automated and have been written concurrently during the iteration by QA with the cooperation of the developers.
• “Peer reviewed”: Developer code reviews completed. I like this concept when paired with feature branching and pull requests.
• “QA complete”: QA testing, automated and exploratory, completed. I like having this as a separate stage-gate, allowing for exploratory testing by QA.
• “Documented”: As needed. There's probably another blog post here around documentation and the agile process, but I will leave that alone for a while.

Now there is little ambiguity as to where a story stands when using this terminology. I may throw another level in there, customer complete, when the customer signs off on the newly developed feature. Thoughts? How do you communicate when a story is complete?

Understanding the power of isolated object testing

I spent some time last night watching J.B. Rainsberger's excellent InfoQ presentation entitled "Integration Tests are a Scam". I've been recently contemplating why I favor unit testing (or as J.B. calls it, isolated object testing) over integration testing and I found his presentation particularly relevant. I highly recommend watching it. He also has a series of blog entries that support the presentation (Parts 1, 2, and 3). I also recommend reading those. They're truly gems.

 

Testing to prove basic correctness

I find many developers using integrated tests as a way to prove the basic correctness of the class or system under test. J.B. writes that "While integration tests offer value in other contexts, too many programmers use them to show basic correctness, and when they do that they waste a tremendous amount of time and effort." Integrated testing can be used within a project (I'm personally fond of acceptance testing), but integrated testing should not be used to prove basic correctness of your code. Focused, isolated object tests (aka unit tests using test doubles) should be used for this endeavor. If you discover behavior that a collaborator demonstrates and you have not accounted for in your isolated object tests, you should mimic this behavior in your test doubled collaborator contracts. You want to cover as much of your code with isolated object tests. We'll talk more about contract tests later.

J.B. mentions using an integrated test to learn about how a collaborator might support its contract, but that this integrated test is not included in the basic correctness test suite. I'm wondering if there isn't some other test suite here that we could use to keep integrated tests that support our learning the runtime and external dependencies. This test suite would be run periodically, but is not part of the whole continuous integration process of building a software system. Need to noodle on this more.

 

Isolated object testing

J.B. states in his aforementioned presentation that he does not use the term "unit testing" and instead favors a more focused term of "isolated object testing". He makes a point to call out the isolated word; these tests isolate the class under test by using test doubles to stub or mock the collaborators of the class under test. These tests focus on a single object and a singular behavior. Any collaborations are realized using test doubles.

I tend to agree that the phrase "unit testing" is a weak phrase describing the type of testing that we use to drive design of a single class and a single method. These tests focus on the responsibility of a single method of a single class. These tests also help you focus on understanding the contracts of collaboration between this class under test and its collaborators. This is really where the design part of the equation comes in. Classes that are hard to test are screaming out that the design is wrong. The tests are saying you have too much responsibility in the class under test or there are too many collaborators. Using your fast isolated object tests, you can engage in a reactive design activity, moving responsibilities out of the class under test, create new collaborators, and other design changes to make .

 

The collaboration contract

J.B. rightfully makes a big deal about contracts (aka interfaces) to explicitly specify collaboration protocols between objects. He mentions that in Domain-Driven Design, popularized by the Eric Evans book of the same name, three concepts can be used to express a domain-driven model: Values, Entities, and Services. All Services should have contracts and those contracts manifest themselves as interfaces. By specifying interfaces, we explicitly declare the protocol supported by each interface implementation. Constraining ourselves to interface types when specifying collaborations results in looser coupled systems, which is considered a "Good Thing". When engaging in isolated object testing, J.B. details the concept of collaboration tests and the closely associated contract tests. This concept of collaboration tests and contract tests is something new to me and is a big reason the presentation was so valuable in my eyes.

 

Collaboration tests and contract tests

After watching the presentation, I seem to be doing a pretty good job of using collaboration tests, but I'm not making the association to contract tests. Collaboration tests prove that the client interacts with its collaborators correctly; the client sends the correct messages and message arguments to the collaborator and appropriately handles all outputs from the collaborator. This is traditionally what I have used mock objects for and that seems to be what J.B. is arguing one should do for collaboration testing.

Contract testing, on the other hand, deals with testing that an interface implementation accurately respects the interface it is implementing. Does the implementor support the contract it declares to support? I haven't typically written these types of tests, but I'm going to start. Interestingly, I don't see a lot of this sort of testing in the wild. The part I really dig about contract tests as Rainsberger explains them is that they can be reused across interface implementations. He uses the List interface and two of its implementations, ArrayList and LinkedList, and details how to use implementation inheritance in the tests to DRY up your contract tests.

It really gets interesting when he declares that for every collaboration test that implies that a test double behavior, there better be a corresponding contract test that demonstrates that the interface implementation(s) actually does support that behavior. Same goes for values returned: if a test double returns a value in a collaboration test, there should be a contract test that demonstrates that the real implementation(s) does actually return that value. This is where I think selective, integrated learning tests can help you discover how your classes on the edges of a system may act when integrated to real external dependencies. But again, those integrated tests are not providing basic code correctness semantics. They're in your project to help you learn, but are not part of the isolated object tests suite. Don't lump them with your isolated tests and don't run them as part of your code/update repos/run tests/commit cadence.  This learning test suite should be run periodically during the day, but not part of the CI build process.

The death spiral of integrated testing for proving basic code correctness

Both Rainsberger's tutorial presentation and his blog postings go into great detail to the fallacy of using integrated testing to prove code correctness. I won't rehash what he has to say about it. My conclusions that I draw from his material are:

• Focus on isolated object tests and isolation techniques and learn how to do collaboration and contract testing.
• If you need to learn, by all means, write an integrated test. If you need to reproduce a defect, initially write it as an integrated test. However, in all cases, take what you have learned and replicate that in your isolated object test suite.  Replicate the behavior you discovered in the integration test(s) with test doubles.  This will ensure that your isolated object tests stay true to the behavior of the integrated system.
• Don't get sucked into the downward death spiral of using integrated testing to guide your test-driven design efforts. It will only cause you pain.

J.B.'s series on Integrated Tests are a Scam can be found here.

 

It's all about the conversations!

More pondering as I contemplate my previous consulting gig.  This time, I'm considering requirements discovery.  Pre-agile, people would write large requirements documents, hoping to document all the requirements needed for the developers to build a system that would satisfy the customers.  Unfortunately, this view of discovering all the requirements ahead of time is awfully naive.  Things change.  Requirements go undiscovered.   Requirements that are captured are not thought out as well as we would like and when it comes time to implement the requirements in software, the requirements don't make sense or are plain wrong.  Thus, large efforts to capture all the requirements has some amount of wasted effort.  This effort focuses most of the conversations at the beginning of the project.  Conversations after the requirements document has been written and signed off is discouraged; it's viewed as evidence that there are errors in the requirements document.  There doesn't seem to be any room for learning with requirements documents.

Now we have agile methods and the user story.  User stories are not requirements.  They're a planning tool for the agile team.  They are a statement of value that the business would like built.  There may or may not be acceptance criteria associated with the user story, depending on where the user story is in its lifecycle.  The most important part of the user story is the conversations that need to happen to flesh the story out so the feature value can be realized in the product.  This detail seems to escape a lot of people trying to use user stories to build products.  User stories give me the freedom to have conversations with all interested parties regarding the statement of value.  I learn just-in-time about the requirements as I implement the feature.  The business learns about how those requirements they communicated manifest themselves in a product.  And we both have the freedom to adjust, learning about the product along the way.

The above statements about user stories does not preclude one from using prepared, written documentation to feed the conversation.  My previous gig, we have a lot of federal government documentation regarding reporting and calculation requirements.  These requirements are set in stone by the government.  Yet, there needs to be conversations as to how those requirements will be accomplished across releases and sprints.  This is where the conversations come back to the forefront.

User stories allow me to deliver value to the business or customer in bite-size portions.  Value is delivered in small increments, thus I can deliver these more frequently and solicit feedback from the business or customer.  When I have short feedback loops in place, I can nimbly change my course if I need to.

Agile database modeling

My previous project has me doing a lot of introspection lately.  One of the recurring themes that I have been noodling on has been evolving a data model using agile development techniques.  The applications being built on this project are based on the .NET platform.  The development group is currently using the Database Project template in Visual Studio 2010 but looking to support a database migration process here soon.  The VS 2010 Database Project template does an admirable job of keeping track of all the DDL for your project, but it offers nothing for refactoring your database over time and migrating a production database.  It seems like the template is meant for rebuilding the database from the ground up, thus there is no concept of database schema changes or migrations.

The database will evolve over time.  Development groups should learn how to build their data model incrementally over many iterations and releases.  Database migration tools can be very helpful in your quest to evolving your database over time.  Tools like Liquibase and Rails migrations are very good at supporting this sort of development behavior.  There are tools on the .NET platform that do this sort of thing.

Another thing that has caused quite a bit of headache is the desire of the data group (data architects and DBAs) to try to get out ahead of the developers some ways and build out much more of the data model than the developers need for the current sprint.  We have found that when your data modeling efforts are not driven from user stories that they tend not to align with the efforts of completing the user stories.   Thus developers and data people end up conversing about data model changes that could have been avoided in the first place by waiting for the right time to initiate changes to the data model.  My advice for building an operational data store that one or more applications will be developed on top of: don't try to develop the entire data model upfront.  You will inevitably end up changing the data model to support requirements as they change (hopefully you're using stories to guide your development).  Typically your application developers will have constraints and needs that need to be accommodated in the data model.  As the data person, you need to be communicating with them and working from user stories.  Ideally, the data people are part of the project team and are dedicated to the agile process.  We really did not have that on this project.  I think that caused issues and slowed us down a bit.

Refactoring Databases should be required reading for development groups.

 

Tweaking your user story mapping efforts

I had a great day leading a business group through a user story mapping session at my current client.  Story mapping is technique that Jeff Patton has popularized for giving your product backlog some structure.  I've done a couple of these user story mapping sessions with pretty good success.  Today, we changed up a couple of things with the session and saw some good results.  Thought it might be worth a blog posting.

First, identify your high-level activities and lay them out across a wall.  We used large Post-It sheets and attached one high-level activity to each sheet.  By doing this, we could move activities and associated tasks around the room, allowing us to rearrange priorities easily.

Next, give the customer/business group five (5) minutes to come up with as many tasks as they can think of for each high-level activity.  Time-boxing the effort keeps you on a regular cadence.  Post the task Post-Its on the large Post-It sheets in no particular order.  Move from one activity to the next, spending the same amount of time on each.  Don't worry about duplicate tasks or the prioritization of the tasks.  You'll come back to these, culling and prioritizing the tasks associated with each activity.

After harvesting the tasks for each activity, go back to each activity and cull out the duplicate tasks and prioritize the tasks according to Jeff Patton's story mapping technique.  We spent 20 minutes on each activity and were able to get a backbone of tasks defined, with other non-core tasks associated with the activity.

Another technique for ensuring that high priority tasks percolate to the walking skeleton row of the story map is to give the business people sticky dots to place on the tasks that they think are core.  We had our business folks put their initials on the sticky dots so we knew who voted up the task.  The dots stand out on the story map and the business really liked using this prioritization technique.  Having the initials on the dots gives you added information regarding who is connected to what stories.

Now that the tasks are prioritized, you can walk your story mapping and talk about it with your customers/business people.  Walking the story map ensures that the ordering of the activities and tasks makes sense and nothing has been missed.  By tweaking our story mapping session today, we were able to keep everyone in the business group engaged and the conversations flowing.

Anatomy of a successful large agile project

I recently had a conversation with a colleague of mine at a company where I’m currently consulting.  We’ve been trying to bootstrap a collection of projects using an agile development process and associated software craftsmanship behaviors.   We have had mixed success to date.  Frustrated with the progress, my colleague asked me to enumerate what I felt were the success factors on the WestlawNext project that I had recently participated on.

I worked at Thomson Reuters from January 2008 to August 2010, during the initial releases of the WestlawNext project.  I was with the project from the beginning of its software development; the product development group had been working on the inception of the WestlawNext project for a couple of years prior.  WestlawNext was a very large project.  Hundreds of people were involved and millions of dollars were spent to build the next generation legal and regulatory research tool.  A lot was riding on this product.  It had to be a success—there was no option for failure.   The following themes are what I feel made this project a success.

 

Attitude

Now that I’ve had time to ponder my WestlawNext experience from afar, I think the number one reason for its success was attitude.  This was an audacious effort to build a new legal research tool in two years time with the number of people involved using an agile software development process.

But from the very beginning, a “can-do” attitude was instilled in the group that we would succeed.  We were going to “knock the ball out of the park” with this product.  There was never a thought that this thing might fail.  Many concerted efforts were made to continually propagate this attitude throughout the participants of the project.  Project tee-shirts, baseball trading cards, raffles, and summer socials were utilized to promote this team spirit.  This infectious attitude allowed us to overcome obstacles that would probably derail other projects.  People were willing to take responsibility for their work and put in the effort over and above the call of duty time and time again.

Communication

Communication is one of the most important functions of a software development project.  Large projects are very susceptible to communication breakdowns as the number of people increase.  We tried to minimize these breakdowns by favoring face-to-face communication as much as possible.  We were encouraged as software developers to pair program.  Designers were encouraged to work directly with developers on styling concerns.  We were encouraged to collaborate together when tough problems arose.  No GoToMeeting.  No conference calls.  Face-to-face conversations.

We were extremely lucky to be able to co-locate almost everyone on the project in three areas of the Thomson Reuters facility in Eagan.  When I mean everyone, I mean business people, vice presidents, directors, testers, designers, managers, coaches, and software developers.  This is one of a few places that I have consulted that have had the luxury of co-locating people in common areas.

 

Leadership

WestlawNext benefited from strong leadership that what 100% dedicated to the project.  No other obligations—they were focused solely on the development on the new product.  Our leaders were also quite familiar with agile software development process.  Some of them had come from other agile projects, both within the company and from outside.  They didn’t have to start from square one and many already knew the key behaviors of the process.  A few of them were software developers at one time (or still are). This is refreshing from a developer’s point of view.  They understand what it takes to build software; they’ve been in the trenches.

There is one moment in particular that I am very fond of.  I was working on a tough networking issue with some .NET code that we had provided another group.  We were throwing spurious socket closed exceptions, but it didn’t happen all the time and it seemed to occur only when the server load increased.  Our senior director, one of our leaders, was helping triage the issue and participating in our root cause analysis.  This senior director had technical chops and was quite proficient at network analysis.  He rolled up his sleeves and got right in and loved being able to help solve the issue.  We did solve the issue; it turned out to be tied to a deprecated thread-local storage API in .NET.  That leader earned a ton of my respect that day.

 

Testing

Testing is paramount to building a quality software product.  The WestlawNext project embraced testing like I have never seen before in my career.  We evolved our designs with unit testing.  We used integration testing to ensure that software components were wired together correctly.  Acceptance testing ensured that features did not regress in future iterations of development.  Load and performance testing was continuously run in an effort to tune the overall product.  Beta testers were allowed to play with the software well in advance of its initial release date, ensuring that it satisfied the customer.

All of this testing allowed us to build tight feedback loops, giving us near-instantaneous data on the health of our growing and evolving product.  The suites of tests infused confidence within the project group; we knew exactly how the software performed at specific load levels.  I cannot fathom working on a software development project that does not fully embrace the aforementioned levels of testing.

 

Conclusion

In conclusion, I’m starting to realize that the WestlawNext project may have been one of those rare moments where everything came together in near perfect harmony to produce a great product.  As I have moved on from Thomson Reuters, I yearn to replicate a similar experience at my other clients.  My current engagement only reinforces the fact that every software development project takes a different path to success, and some may never make it to the end.

 

 

 

Using argument matchers in EasyMock and mockito

I spent some quality time with EasyMock today while coaching some developers on building tests around an existing codebase. We were mocking a dependency of our SUT and the method call that we wanted to mock had two parameters. When we wrote the expectation, we used the anyObject() matcher for the first parameter and tried to perform an exact match on the second parameter, a Boolean. Here is an example of what we were trying to do in EasyMock (NOTE: we statically imported EasyMock so we could reduce the amount of code noise by not prefacing our expect, matcher, replay and verify invocations with EasyMock.):


expect(mockObject.retrieveSomething((String) anyObject(), false)).andReturn(someObject);


Unfortunately, EasyMock and mockito do not like this. They both want you to use matchers for all parameters if you use matchers for any parameters. However, the two libraries react quite differently when this situation occurs. EasyMock complains with a somewhat confusing message that at first blush makes it seem like we declared the expectation for multiple invocations. It really threw us off for a while (at least an hour) trying to figure out what was wrong with our expectations. Here is how we fixed it in EasyMock:


expect(mockObject.retrieveSomething((String) anyObject(), eq(false))).andReturn(someObject);


Mockito does a much better job of stating that when you use a parameter argument matcher in your expectation, you have to use parameter argument matchers for all of the parameters of the method call participating in the expectation. I find it interesting that mockito retains the behavior of EasyMock (mockito is a fork of EasyMock) with regards to argument matching, but mockito improves on the error messaging when something goes wrong with the mock object setup. Further reinforces my decision to forego EasyMock in favor of mockito.

Getting a handle on code quality with Sonar

I've been working with a client recently who uses Sonar, an open source code quality management platform hosted at Codehaus. I'm thoroughly impressed with this tool. If you want to see what the tool can do without investing the time in installing it, take a look at the various screencasts. They also have a live version of Sonar up so you can play with it without installation.

We've been working on getting unit tests built around a legacy code base and Sonar has been a big help in identifying classes that are the biggest code coverage offenders. We used the Clouds feature, a word cloud that weights the class names in the cloud based on code coverage and complexity. The less test coverage on the class and/or the more complex the class, the larger the weight of that class name word in the word cloud. It really helped us focus on where to direct our testing efforts.

I have yet to get this tool up and running in one of my own projects, but things are finally starting to simmer down now with consulting and training activities that I hope to focus on building out a CI environment using Hudson and hooking in Sonar to that environment. Stay tuned.

Learn IDE key mappings while pair programming

I'm a big believer in keeping the hands on the keyboard and minimizing the amount of mouse use when coding. Therefore, I tend to spend some time learning the keymappings for the IDE I am using. I'm most proficient with IntelliJ IDEA but I've also learned a good portion of the keymappings in Eclipse and Visual Studio+ReSharper.

I've been doing a fair amount of pair programming lately on my current gig, a good thing. One behavior that we have been practicing while pair programming is gently forcing the driver of the pair to use keymappings to activate various actions within the IDE. Neal Ford describes this in his book The Productive Programmer. Practicing this behavior while pair programming has proven very valuable to increasing our productivity while in the code base. Neal mentions the KeyPromoter plugin for IntelliJ; I've tried this plugin and I haven't become a big fan of it. Much easier if your navigator keeps you honest.

I've done this before with pair programming while working at Identix in 2004 with Hans Loedolff. Hans knew all the IntelliJ keymappings and could type at about 90 words per minute. He was an excellent pair programmmer.

Managing stories electronically

I've been involved in this agile project at my client for some time and for the past three months or so I've been working in a group that favors a low fidelity mechanism for managing their stories--Post-It notes on a board. The stories, tasks, and acceptance tests are all written on Post-It notes. We are required to keep the stories and tasks in Team Foundation Server (TFS) also, but the group manages the iteration's work from the Post-It notes. The TFS eScrum template is not a particular good system for managing stories and tasks, so we tend to stick to the low fidelity method.

I really dislike this behavior. The Post-It notes are difficult to keep stuck to anything for the entire iteration. They drop off the board and mysteriously get lost. The Post-It notes are typically written in many different handwritings. There's no consistency in display of stories and tasks. Some of the Post-It notes are almost unreadable. The format of the Post-It notes is not consistent. It's difficult to write much on a single Post-It note. Reporting, well...

I'm a big believer in managing your stories and tasks electronically. Use the Post-It notes during iteration planning to facilitate teamwork and collaboration, but don't leave the stories, acceptance tests and tasks in that form for any amount of time. Get them into a tracking system that allows you to easily manage the stories and tasks for the iteration. I'm a big believer of a dashboard view that gives you a high level overview of all the iterations stories and their progress in a single view.

Powered by Zoundry Raven

Measuring value during an iteration

I'm currently working on a large agile project where we measure task burn-up. Our user stories are decomposed into individual tasks and the tasks are estimated. When a developer or a development pair completes a task, they mark that task complete in the tracking system. A daily chart is then generated by the tracking system that displays a line graph of units of tasks completed in the current iteration. The cumulative total task units from the estimates is another line on the graph, and it typically stays horizontal across the entire iteration. Up until recently I hadn't given much thought to these generated charts.

Now I'm wondering what value these charts actually provide. If management is interested in seeing that people are busy working and completing tasks, then these charts are spot on. They definitely will show the amount of work completed during the iteration and when it's completed. But really, is that metric all that important? I tend to say no. One issue off the top of my head that I have seen happen in our group is the completion of tasks on stories, but not fully completing the stories. For whatever reason, our user stories are drifting from one iteration to the next, never reaching the point of completion. Therefore, I conclude that we're keeping ourselves busy, but not adding any value to the overall product.

I want to know how much value I have built into the product. To measure that quantitatively, you need to measure user story burn-up. If you measure user story burn-up, you will focus the development team on completing stories. I think the emphasis needs to be on the user story; the task is a planning construct that just helps us decompose the story into units of development that can be worked on concurrently by a number of developer pairs. I don't know if I really care about task estimates anymore either. I'm drifting towards estimating at the user story level only.

Using ArgumentMatcher to capture indirect output arguments

Yesterday, I posted a blog entry about performing assertions directly in your ArgumentMatcher implementations.  I wanted to close the loop on my previous usage of the ArgumentMatcher.  After some discussion with the main committer to mockito, Szczepan Faber, it seems that ArgumentMatcher should only be used to capture the indirect output argument, making it accessible to the test code. Once the test code has the indirect output argument available to it, you can then assert on it to your heart's delight. Szczepan recommended not embedding the assertions directly in the ArgumentMatcher implementation.  More information can be found here.  Seems like a good approach. 

.NET Base Class Library types not friendly to mocking with MoQ

I spent some quality time this week driving a design implemented in .NET using unit tests.  It was an enlightening experience and gave me a good glimpse at the quality of the BCL design.  I'm not impressed. We had issues trying to mock System.Net.WebRequest and System.Net.WebResponse and its decendents.  It seems that Microsoft prefers to expose abstract base classes, instead of using interfaces for this.  The aforementioned types are not interface types; they're abstract classes.  We've been using MoQ, a fine mocking framework, at the client I've been working at and I tried to create mock implementations of the System.Net.WebRequest and System.Net.WebResponse types.  No go--these types do not have public constructors.  The non-virtual, non-abstract methods also give MoQ headaches (good write up here about why methods of these abstract class types have to be explicitly decorated with virtual if you want to mock them).  Personally, I'm more fond of exposing interface types, thus alleviating the problem all together.  Very frustrating.  We did continue to unit test and utilized mocks for this, but it forced us to wrap these BCL types with our own types where we could control the virtual decoration.  Yuck!!