Anatomy of a successful large agile project

I recently had a conversation with a colleague of mine at a company where I’m currently consulting.  We’ve been trying to bootstrap a collection of projects using an agile development process and associated software craftsmanship behaviors.   We have had mixed success to date.  Frustrated with the progress, my colleague asked me to enumerate what I felt were the success factors on the WestlawNext project that I had recently participated on.

I worked at Thomson Reuters from January 2008 to August 2010, during the initial releases of the WestlawNext project.  I was with the project from the beginning of its software development; the product development group had been working on the inception of the WestlawNext project for a couple of years prior.  WestlawNext was a very large project.  Hundreds of people were involved and millions of dollars were spent to build the next generation legal and regulatory research tool.  A lot was riding on this product.  It had to be a success—there was no option for failure.   The following themes are what I feel made this project a success.

 

Attitude

Now that I’ve had time to ponder my WestlawNext experience from afar, I think the number one reason for its success was attitude.  This was an audacious effort to build a new legal research tool in two years time with the number of people involved using an agile software development process.

But from the very beginning, a “can-do” attitude was instilled in the group that we would succeed.  We were going to “knock the ball out of the park” with this product.  There was never a thought that this thing might fail.  Many concerted efforts were made to continually propagate this attitude throughout the participants of the project.  Project tee-shirts, baseball trading cards, raffles, and summer socials were utilized to promote this team spirit.  This infectious attitude allowed us to overcome obstacles that would probably derail other projects.  People were willing to take responsibility for their work and put in the effort over and above the call of duty time and time again.

Communication

Communication is one of the most important functions of a software development project.  Large projects are very susceptible to communication breakdowns as the number of people increase.  We tried to minimize these breakdowns by favoring face-to-face communication as much as possible.  We were encouraged as software developers to pair program.  Designers were encouraged to work directly with developers on styling concerns.  We were encouraged to collaborate together when tough problems arose.  No GoToMeeting.  No conference calls.  Face-to-face conversations.

We were extremely lucky to be able to co-locate almost everyone on the project in three areas of the Thomson Reuters facility in Eagan.  When I mean everyone, I mean business people, vice presidents, directors, testers, designers, managers, coaches, and software developers.  This is one of a few places that I have consulted that have had the luxury of co-locating people in common areas.

 

Leadership

WestlawNext benefited from strong leadership that what 100% dedicated to the project.  No other obligations—they were focused solely on the development on the new product.  Our leaders were also quite familiar with agile software development process.  Some of them had come from other agile projects, both within the company and from outside.  They didn’t have to start from square one and many already knew the key behaviors of the process.  A few of them were software developers at one time (or still are). This is refreshing from a developer’s point of view.  They understand what it takes to build software; they’ve been in the trenches.

There is one moment in particular that I am very fond of.  I was working on a tough networking issue with some .NET code that we had provided another group.  We were throwing spurious socket closed exceptions, but it didn’t happen all the time and it seemed to occur only when the server load increased.  Our senior director, one of our leaders, was helping triage the issue and participating in our root cause analysis.  This senior director had technical chops and was quite proficient at network analysis.  He rolled up his sleeves and got right in and loved being able to help solve the issue.  We did solve the issue; it turned out to be tied to a deprecated thread-local storage API in .NET.  That leader earned a ton of my respect that day.

 

Testing

Testing is paramount to building a quality software product.  The WestlawNext project embraced testing like I have never seen before in my career.  We evolved our designs with unit testing.  We used integration testing to ensure that software components were wired together correctly.  Acceptance testing ensured that features did not regress in future iterations of development.  Load and performance testing was continuously run in an effort to tune the overall product.  Beta testers were allowed to play with the software well in advance of its initial release date, ensuring that it satisfied the customer.

All of this testing allowed us to build tight feedback loops, giving us near-instantaneous data on the health of our growing and evolving product.  The suites of tests infused confidence within the project group; we knew exactly how the software performed at specific load levels.  I cannot fathom working on a software development project that does not fully embrace the aforementioned levels of testing.

 

Conclusion

In conclusion, I’m starting to realize that the WestlawNext project may have been one of those rare moments where everything came together in near perfect harmony to produce a great product.  As I have moved on from Thomson Reuters, I yearn to replicate a similar experience at my other clients.  My current engagement only reinforces the fact that every software development project takes a different path to success, and some may never make it to the end.

 

 

 

DevJam Test Driven training December 2-3, 2010

Photos and video from today's Test Driven class at DevJam U.

Kaleidoscope diff tool for Mac OS X

Found a really interesting new diff and merge tool for the Mac: Kaleidoscope.  Native app, integrates with Versions, Cornerstone, and the command line.  Looks promising.

 

Autofixture: A generic Test Data Builder implementation for .NET

Just came across a Test Data Builder implementation for .NET, Autofixture.  The Test Data Builder pattern has become quite popular recently since it was mentioned in Growing Object-Oriented Software, Guided by Tests.  I've used the pattern before, but I've always built the builder implementations by hand.  This implementation looks really promising.

Great video on what motivates us

Great video on drive and motivation. Love the whiteboard drawings. Spend 10 minutes watching this video.



Loving my new Magic Trackpad

Just received my Magic Trackpad from Apple today.  Very impressed after a bit of use with it.  Love all the different gestures that you can map to.  After using this, a mouse is going to seem awfully archaic.

Brief history of mock objects

A brief history of mock objects, a story told by Steve Freeman. Good read as to the "why" of mock objects and how Hamcrest, jMock, and others came about.

http://www.mockobjects.com/2009/09/brief-history-of-mock-objects.html

Steve Freeman on sustainable TDD

Excellent presentation by Steve Freeman on sustainable TDD.  Lots of great tips for making your unit tests easier to comprehend and maintain.

http://www.infoq.com/presentations/Sustainable-Test-Driven-Development

 

 

Practical styles of pair programming

Excellent blog on pair programming.

http://blog.xebia.com/2010/05/09/practical-styles-of-pair-programming/

A quote from the blog entry:

"No you're not faster on your own, you're just creating more crap for your colleagues to puzzle over and eventually delete. The code you write alone sucks. That guy that is getting on your nerves is trying to tell you (clumsily) that your code sucks, try to listen to him and you'll turn into a better programmer." 

Have you encountered one or more of these styles?  How many developers are pair programming these days?

 

Tool coming to Xcel Energy Center

Tool is coming to the Twin Cities this summer!

Tool concert on July 1, 2010

Tickets go on sale this Saturday.

Uninstalling Mac developer tools

If you ever need to uninstall the Mac OS X developer tools, you do it with one command:

sudo /Developer/Library/uninstall-devtools --mode=all

Found this advice here.

Mac OS X key bindings

I'm trying to make my Xcode environment a bit more friendly and thus I'm investigating key bindings. A good article that summarizes how to change key bindings system-wide.

http://www.erasetotheleft.com/post/mac-os-x-key-bindings/

Unfortunately, this doesn't seem to work with Xcode. My original investigation started here.

Test-Driven in Groovy presentation in the bag

Joe Muraski and myself did a Test-Driven in Groovy workshop at GR8 in the US conference here in Bloomington, MN this past Friday. If you're interested in the materials (presentation, demo app), you can get it all right here. The demo consists of a Java web app with lots of opportunity for refactoring, Groovy unit tests, and Cucumber (via cuke4duke) acceptance tests.

iPhone presentation

What a great turn out for the iPhone presentation last night. Thanks to everyone that came out. I've updated the Google Groups page for the presentation with links to blogs and other interesting information pertaining to iPhone platform development.

http://groups.google.com/group/jamsessions/web/10-04-07-iphone-os-the-next-killer-platform

A huge thanks goes out to my co-presenter, Bob McCune. The SenateJam demo was a big hit.

Preparing for DevJam Jam Session iPhone presentation

I've been spending some time working on an iPhone development presentation that Bob McCune and I will co-present at DevJam's Jam Session this coming Wednesday. I was looking at job trends at indeed.com--iPhone is #3 on the list of top job trends. Here's the graph:

iPhone Job Trends

iPhone jobs

Wow! Talk about growth. Looking forward to the presentation.

First impressions of the new iPad

What a beautiful device! A little bit heavy to hold in one hand for an extended period of time. Smudges from finger taps and swipes are more noticeable on this device, especially when reading text. I'll have to investigate a protective film and see that makes any difference.

The processor in this device is very fast; the device is very snappy with animations, games, and video. I can see game makers really going after this thing. EA seems to have a couple of games out for the iPad (I tried Scrabble and Need for Speed). Game control was very good for Need for Speed.

Transferring .epub files to your iPad

Transferring .epub files purchased outside of iTunes seems to be very easy. Download the .epub file to your computer. Open up iTunes and select File -> Add to Library... menu item. From the file chooser, select the .epub file that you want to transfer to your iPad. Click Choose and the eBook file will be added to your Book Library in iTunes 9.1. Sync as usual and the book will show up in iBooks.

Nice Mercurial tutorial by Joel Spolsky

Joel Spolsky (of Joel on Software) wrote a very entertaining and thorough tutorial on Mercurial, a distributed version control system (DVCS). The tutorial has a Subversion re-education pre-tutorial for those of us with Subversion brain damage. Well worth a look. Sadly, Joel will no longer be blogging, citing the need to focus on his growing software company. His musings on software development will be missed.

Mockito's @InjectMocks annotation does reflection-based DI

I've been using @InjectMocks heavily lately since it came out in version 1.8.3 of mockito. The javadocs for this annotation state that it uses setter injection to inject its dependencies. Being a lazy developer, I was writing a unit test yesterday at work and forgot to write the setter methods for a couple of dependencies on the SUT that I was testing. Lo and behold, the test passed and all mock verifications were satisfied. Very confused, I went back to some other unit tests and their SUTs and removed the setter methods. In all of my cases, the tests continued to pass. A quick note to the mockito list confirms what I discovered--@InjectMocks actually is using a reflection-based DI scheme, not unlike what Spring does when you annotate collaborator fields in a Spring bean with @Autowired and @Resource. Very cool feature, as this further reduces code noise. Here is the discussion on Google Groups.

Cool features in Cucumber, part 1

My Cucumber adventure (via cuke4duke) continues. I have been busy writing features and building out step definitions in Groovy. Groovy absolutely rocks for this sort of thing. I'm doing some more elaborate Cucumber features (at least in my novice head) and came upon this blog posting. Very handy stuff. More related to the Ruby version of Cucumber, but most of the concepts transfer over to cuke4duke.

PeepCode screencast: Use the Cucumber

I've been interested in acceptance testing tools recently and Cucumber has been at the top of my list. PeepCode recently released a screencast on Cucumber. I've viewing it right now; the screencast is excellent and a nice introduction to Cucumber. I'm thinking of using it on a Grails web app. The screencast is based on a Rails 2 web application, which Cucumber seems to integration seamlessly with. Supposedly Cucumber is technology agnostic and works against any sort of web application.

CoverScout for retrieving iTunes album art

I recently purchased the MacHeist nanoBundle, a collection of Mac OS X applications and utilities. One of the utilities included in the bundle is CoverScout, a utility that will allow you to browse your iTunes library and retrieve hard to find album art. I have a fairly large collection of music and iTunes has been pretty average in its attempts to find album art. CoverScout will retrieve album art from many different sources and allow you to select which image to use. The images are also rated by quality. Very cool utility.

New annotations in mockito 1.8.3

Mockito 1.8.3 was recently released and I got a chance to use it today. There are a couple of new annotations available in this release: @Captor, @Spy, and @InjectMocks. I was able to use @Captor and @InjectMocks today. Even after just a few minutes with these two annotations, I'm sold. Very cool enhancement. I spent some quality time with EasyMock last week, and there's nothing easy with EasyMock. Ugh!! If you're hip to using mock objects in your Java unit testing efforts, you really should look at mockito these days. Details about @Captor and @InjectMocks follows.

@Captor

This annotation will automatically create typed argument captors (org.mockito.ArgumentCaptor<T>) in your unit tests. Argument captors are essential in verifying indirect outputs to your mocked collaborators.

public class Test{
    @Captor ArgumentCaptor<Foobar> foobarCaptor;

    @Before
    public void init(){
       MockitoAnnotations.init(this);
    }

    @Test 
    public void shouldDoSomethingUseful() {
       //...
       verify(mock.doStuff(foorbarCaptor.capture()));
       Foobar capturedFoobar = foobarCaptor.getValue();
       assertEquals("foobar", capturedFoobar.getName());
    }
 }

@InjectMocks

Automatically injects mocks by type using setter injection. Constructor injection is not currently available, but if you want to provide a patch, the mockito team will gladly consider your contribution. I'm actually more interested in reflection-based injection, similar to what Spring uses when annotating dependency fields using @Autowired. Having your unit tests inject dependencies via reflection would help me avoid the set* methods on the implementations. I may have to play with this a bit.

public class FooBarManagerTests  {

       @Mock private FooDependency mockFoo;
       @Mock private BarDependency mockBar;
       @InjectMocks private FooBarManager manager = new FooBarManagerImpl();

       @Before 
       public void initMocks() {
           // Initializes all mocks and then injects those mocks into the FooManager instance.
           MockitoAnnotations.initMocks(this);
       }

       @Test 
       public void shouldDoSomething() {
           manager.doSomething();
           verify(mockFoo).doSomethingToFoo(any(String.class));
           verify(mockBar).doSomethingToBar(any(Integer.class));
       }
   }

Gerard Meszaros "From Concept to Product Backlog" talk

I just finished up an InfoQ presentation by Gerard Meszaros discussing how to go from the conceptualization of a project to the manifestation of a product backlog. I really enjoyed the presentation and it hits on some gray areas of current agile implementations that I have worked on. Highly recommended.

Adopting Agility with XP, Scrum and Lean Thinking - Powered by RegOnline

David Hussman is leading an agile course next week. Still room for participants to register.


Adopting Agility with XP, Scrum and Lean Thinking

DevJam course offerings for February-March 2010

David Hussman recently posted the new DevJam course offerings for February and March 2010. I'll be leading the Introduction to Groovy, Introduction to Grails and Test Driven courses. The Test Driven course is open to Java and .NET developers (.NET examples in C#). Detailed information can be found at http://www.devjam.com/courses-and-training/index.php. Seating is limited to about 12 participants per course.

IntelliJ IDEA 9 as an Adobe Flex IDE

If you're on the fence about paying for a license for IntelliJ IDEA, here is one more reason to justify the cost of the license. Adobe Flex. I've spent just a little time with it and the Flex 4 Beta 2 SDK and I must admit, IntelliJ as a Flex IDE rocks. Much better than Flex Builder 3. You don't get a Design View with IntelliJ, but I never used it in Flex Builder and it seems of dubious value.

I can't comment on how it stacks up against Flash Builder 4, but Flex development in IntelliJ is a joy. I'll post more as a I get a chance to use it some more.