Agile database modeling

My previous project has me doing a lot of introspection lately.  One of the recurring themes that I have been noodling on has been evolving a data model using agile development techniques.  The applications being built on this project are based on the .NET platform.  The development group is currently using the Database Project template in Visual Studio 2010 but looking to support a database migration process here soon.  The VS 2010 Database Project template does an admirable job of keeping track of all the DDL for your project, but it offers nothing for refactoring your database over time and migrating a production database.  It seems like the template is meant for rebuilding the database from the ground up, thus there is no concept of database schema changes or migrations.

The database will evolve over time.  Development groups should learn how to build their data model incrementally over many iterations and releases.  Database migration tools can be very helpful in your quest to evolving your database over time.  Tools like Liquibase and Rails migrations are very good at supporting this sort of development behavior.  There are tools on the .NET platform that do this sort of thing.

Another thing that has caused quite a bit of headache is the desire of the data group (data architects and DBAs) to try to get out ahead of the developers some ways and build out much more of the data model than the developers need for the current sprint.  We have found that when your data modeling efforts are not driven from user stories that they tend not to align with the efforts of completing the user stories.   Thus developers and data people end up conversing about data model changes that could have been avoided in the first place by waiting for the right time to initiate changes to the data model.  My advice for building an operational data store that one or more applications will be developed on top of: don't try to develop the entire data model upfront.  You will inevitably end up changing the data model to support requirements as they change (hopefully you're using stories to guide your development).  Typically your application developers will have constraints and needs that need to be accommodated in the data model.  As the data person, you need to be communicating with them and working from user stories.  Ideally, the data people are part of the project team and are dedicated to the agile process.  We really did not have that on this project.  I think that caused issues and slowed us down a bit.

Refactoring Databases should be required reading for development groups.